-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Modern level set format #52
Comments
Previously discussed here: #27 (comment) I outlined a basic level set file layout there that should work. Most resources are encoded as JSON and refer to other files if any binary data is included in those resources. I think that a manifest of all the files available in the set would be helpful in locating specific resources quickly. It also prevents us from doing too much file system introspection. This folder structure could be zipped and hashed as a unit for versioning.
|
There are sometimes multiple hull and text resources, so it might make sense for them to be directories of numbered files as well. Another thing to mention is that LEDIs often referenced PICT resources by their "name" rather than their ID, but we can always standardize them to use resource IDs when we run them through the converter. I'm not certain what the point of the manifest file is, exactly... it might make it easier to find things from the perspective of the source code, but it places an additional burden on the level designer that I don't think should be necessary. I like the idea of zipping sets up into archives, but the game should be able to read uncompressed level directories as well for ease of level editing/testing purposes as well. Another topic for discussion is how level loading/look-up will work. Having to peek into a long list of compressed sets, even in a designated level directory, is probably performance prohibitive. I've been suspecting for awhile that Avara may need to maintain some sort of level look-up cache, linking whatever identifiers are sent over the network to a particular path on the local drive. Might make for a convenient and performant in-game level searching interface, as well. (Assuming that some sort of asset server is not providing this type of functionality already.) |
The reason I didn't make multiple files for LEDI and HULL is because those resources are all text anyway, so they can just be embedded completely in a JSON file with their metadata without getting very long at all. LEDIs are just some metadata and pointers to level data, HULLs are like 7 numbers or something like that each. But they could easily be separate files. the manifest file can be auto-generated. the intent is to prevent Avara from having to look at the file system at runtime, for example, to tell if there even is a Archiving can be an afterthought... it will probably have more to do with some ultimate level distribution solution later on down the line. I see what you're saying about lookup, and that's something i also want to avoid doing with the file system--so how about if the manifest file had every set? We could put a script in The look-ups can work the same way they do now with the 4 letter descriptor. JSON parsing is pretty cheap. later we can update this file remotely through an update server or something. Kind of a stopgap measure until we get a better method of distribution. What do you think? It isn't ideal for a final product but it gets rid of the binary resource fork files in our repo, PICT parsing code, etc. |
Actually, I've never seen a set with more than one LEDI resource (ID 128)... though multiple "fields" are stored within that one resource. So yeah, I would definitely keep LEDI to a single file named `ledi.json` or even something more clear like `directory.json`.
Custom hulls are typically referenced by their resource ID in level files, or via custom aliases defined within a TEXT resource (typically ID 1000 which I believe is auto-loaded by the game when a level is loaded). That's why I thought it was probably a good idea to separate them out into separate files. But also, consistency is probably a good thing to embrace, even though I don't think using these seemingly arbitrary IDs is very ergonomic. (Tooling, like a command line app, could be used to facilitate this somewhat in the future.)
I understand the manifest can be auto-generated by level conversion tools, my concern was mostly revolving around somebody making new levels.
What you suggest about a global manifest is more or less what I was envisioning, but again it poses problems for people developing new levels. (So, yeah, the manifest needs to be able to update itself outside of VCS.)
Re: the level set and individual level descriptors, we could do it the old way, yes, but I never felt good about that system. It places burden on the level designer to assign unique values, but there's basically no way for an individual to be absolutely sure the values are unique. One of the reasons I was looking forward to an asset server, where IDs could be assigned by that system automatically. Maybe we can generate UUIDv4s for every set and level during the conversion process (and again for future sets via additional tooling) and use those, instead?
In any case, any progress would be good progress. Right now it's almost impossible to make new levels at all, and adding "new old" levels in the game is more painful than it should be. So I don't mean to disparage any suggestion. :) Just trying to spew out all the things that have been rattling around in my brain since... the pavara days, basically, about both the technical and design challenges that need to be overcome.
|
Could the manifest and ledi be combined? It seems like they serve a similar purpose. |
Yeah, having the manifest generated with a script, it could gather up the LEDI and store it right alongside the general index of files no problem. But I think we do want level set specific LEDI files so they're easier to edit for the designers and also hash for versioning. As long as set designers don't have to mess with manifest.json themselves. |
@tra I actually had a similar thought a night or two ago, but forgot to talk about it! I do like the idea a lot. I believe LEDI may even stand for "level directory" which is why I suggested the file could be named I still dislike that the level designer would need to maintain the file whenever they add a new resource, some stuff in particular would feel egregious (like when adding a TEXT 1000 resource to set global variable names for referencing other resources)... but again, tooling can help here. I'm envisioning a command line app for "initializing" new level sets with a basic directory structure, adding new levels to it, and using file system introspection to auto-generate/refresh the manifesty bits. It also occurs to me that maybe the text resources can also just be shoved into the manifest file directly. I'd need to do more research to determine what other "magic" text resource IDs may exist or how Avara loads them. @assertivist I think we're gonna need to do more than hash the manifest for versioning, since any of the resources (particularly the SVGs) may have changed. |
Yeah TEXT and HULL can also just get shoved in there really. You might have to manually populate a struct or so for HULLs. I agree, I think that each file should be hashed individually within the structure, including ledi.json (basically i was saying that the manifest shouldn't replace ledi.json even if manifest.json has a copy of that data in there) |
I really like this idea. In general I'm not a fan of redundant information. Let the existence of the level files drive what is shown in the levelset list. But you might need a level naming convention to drive ordering. For example, order alphabetically unless the levels follow some naming conventions with numbers? |
Ordering (which we could also consider Searching) and display of level sets is definitely a concern when we're talking about converting all of Avara's user content, but I think that's firmly in the category of waiting until we have an ultimate level distribution system--think instead of ordering so that your levels are grouped, having meta tags attached so people can search them out by typing "gzr" for example |
Had this sitting around for a bit and wanted to post it before I forgot again. Here's what I was thinking for a "set manifest" file. (Not sure what to actually name it.)
Note the usage of UUIDs and lack of enables/reserved attributes for levels (which could theoretically still be added later without breaking compatibility, as they could just default to 0). I ultimately decided to put the level IDs inside the level object rather than use them as property names (or hash set keys) in order to maintain the order of levels in the set. It will slow down searching for levels within a set slightly, which maybe would be a problem if a level set had thousands of levels in it. They don't. Looking at this, I'm reminded that during the conversion process we should probably convert \r to \n as well as transcode MacRoman to UTF-8. I'd also suggest trimming level descriptions, which in this example would get rid of the leading \r. (I had put them in there to vertically center the text in the old UI.) The Still unsure on how I'd like hulls to be stored. |
Considering your comments and also looking at the output of the conversion tool, I've revised the interim structure a little bit:
Putting all the data from all sets into one file looked cool, but even with the 36 sets we have it ended up being like 7000 lines. So the Hull data can be included in
|
My first comment/question is whether the filenames would include "levels" in the path? For example Second question, can you accommodate multiple designers in a set? For example, the gzr sets typically have multiple designers. Should that be allowed to be overridden on a level by level basis? |
Yeah it would just have it check for the file in
That is already the case, the example is using the TEXT resource so that those script assignments would apply to every level (since he's the designer of all of them). Typically you would put |
Some other random thoughts regarding levelset formats
|
That image showed up in the space where the level descriptions did, but only if you didn't have a level loaded, IIRC. I'll check and take a screenshot, it could be a feature of the roster menu
We could easily support a credits line... Later on I imagine having a better interface for browsing levels. Data like that could easily be shown there.
I wrote several paragraphs, and then decided that what you should do is go ahead and open an enhancement issue for "a better interface for browsing and loading levels". I know we've had some lengthy discussions in the past about several different ways to make that happen and support searching and tagging like you describe. |
It appeared in place of the Ambrosia SW logo, under very specific conditions. (Freshly opened Avara, opened the level set, did not load a level, cover up the logo with another window, then uncover it.) Personally, I was not planning on supporting this feature. Regarding the README and set tagging, I think this is dependent on how levels will ultimately be distributed and installed. If people are manually downloading and installing asset bundles, a README should probably exist outside the bundle, for example. Secondary organization of sets could be handled from within the "manifest" (a term that is becoming increasingly inaccurate, I think), or possibly as metadata on a centralized asset server. As an example here, if we could validate that a level set belonged to a certain author/guild (because it was uploaded by a particular account), that would be better than trusting whatever the author put in their manifest file. |
With the move away from resource forks, we need a method of storing a level set's drawings (svg), sounds (wav or other format), metadata (jsons of resource data) and shapes (avarabsps).
Avara originally worked by overlaying the level set resource files with its own to allow the level sets to override specific resources in the base game and change shapes or other configuration.
Our solution must also be able to dynamically search for a given resource ID from an Avara base level data file and then from the currently loaded set file.
The text was updated successfully, but these errors were encountered: