Modern level set format

[assertivist]

With the move away from resource forks, we need a method of storing a level set's drawings (svg), sounds (wav or other format), metadata (jsons of resource data) and shapes (avarabsps).

Avara originally worked by overlaying the level set resource files with its own to allow the level sets to override specific resources in the base game and change shapes or other configuration.

Our solution must also be able to dynamically search for a given resource ID from an Avara base level data file and then from the currently loaded set file.

[assertivist]

Previously discussed here: #27 (comment)

I outlined a basic level set file layout there that should work. Most resources are encoded as JSON and refer to other files if any binary data is included in those resources. I think that a manifest of all the files available in the set would be helpful in locating specific resources quickly. It also prevents us from doing too much file system introspection. This folder structure could be zipped and hashed as a unit for versioning.

levels/set_name/
│   manifest.json
│   ledi.json
│   hull.json
└───bsp/
│   │    100.json
│   │    ...
│   hsnd.json
└───hsnd/
│   │   100.wav
│   │   ...
└───svg/
│   │   1000.svg
│   │   ...
│   text.json

[rherriman]

There are sometimes multiple hull and text resources, so it might make sense for them to be directories of numbered files as well. Another thing to mention is that LEDIs often referenced PICT resources by their "name" rather than their ID, but we can always standardize them to use resource IDs when we run them through the converter. I'm not certain what the point of the manifest file is, exactly... it might make it easier to find things from the perspective of the source code, but it places an additional burden on the level designer that I don't think should be necessary.

I like the idea of zipping sets up into archives, but the game should be able to read uncompressed level directories as well for ease of level editing/testing purposes as well.

Another topic for discussion is how level loading/look-up will work. Having to peek into a long list of compressed sets, even in a designated level directory, is probably performance prohibitive. I've been suspecting for awhile that Avara may need to maintain some sort of level look-up cache, linking whatever identifiers are sent over the network to a particular path on the local drive. Might make for a convenient and performant in-game level searching interface, as well. (Assuming that some sort of asset server is not providing this type of functionality already.)

[assertivist]

The reason I didn't make multiple files for LEDI and HULL is because those resources are all text anyway, so they can just be embedded completely in a JSON file with their metadata without getting very long at all. LEDIs are just some metadata and pointers to level data, HULLs are like 7 numbers or something like that each. But they could easily be separate files.

the manifest file can be auto-generated. the intent is to prevent Avara from having to look at the file system at runtime, for example, to tell if there even is a 101.avarahull.json or whatever to read for this set--when I was prototyping, file system introspection in cross-platform C++ was extremely painful in various ways.

Archiving can be an afterthought... it will probably have more to do with some ultimate level distribution solution later on down the line.

I see what you're saying about lookup, and that's something i also want to avoid doing with the file system--so how about if the manifest file had every set?

We could put a script in bin/ that generates it for all the folders in levels/. That takes most of the burden off of ALD, and the same script can give some basic validation feedback. You could still go in there and add stuff by hand as well if you were desperate not to install python.

The look-ups can work the same way they do now with the 4 letter descriptor. JSON parsing is pretty cheap. later we can update this file remotely through an update server or something. Kind of a stopgap measure until we get a better method of distribution.

What do you think? It isn't ideal for a final product but it gets rid of the binary resource fork files in our repo, PICT parsing code, etc.

[rherriman]

Actually, I've never seen a set with more than one LEDI resource (ID 128)... though multiple "fields" are stored within that one resource. So yeah, I would definitely keep LEDI to a single file named `ledi.json` or even something more clear like `directory.json`. Custom hulls are typically referenced by their resource ID in level files, or via custom aliases defined within a TEXT resource (typically ID 1000 which I believe is auto-loaded by the game when a level is loaded). That's why I thought it was probably a good idea to separate them out into separate files. But also, consistency is probably a good thing to embrace, even though I don't think using these seemingly arbitrary IDs is very ergonomic. (Tooling, like a command line app, could be used to facilitate this somewhat in the future.) I understand the manifest can be auto-generated by level conversion tools, my concern was mostly revolving around somebody making new levels. What you suggest about a global manifest is more or less what I was envisioning, but again it poses problems for people developing new levels. (So, yeah, the manifest needs to be able to update itself outside of VCS.) Re: the level set and individual level descriptors, we could do it the old way, yes, but I never felt good about that system. It places burden on the level designer to assign unique values, but there's basically no way for an individual to be absolutely sure the values are unique. One of the reasons I was looking forward to an asset server, where IDs could be assigned by that system automatically. Maybe we can generate UUIDv4s for every set and level during the conversion process (and again for future sets via additional tooling) and use those, instead? In any case, any progress would be good progress. Right now it's almost impossible to make new levels at all, and adding "new old" levels in the game is more painful than it should be. So I don't mean to disparage any suggestion. :) Just trying to spew out all the things that have been rattling around in my brain since... the pavara days, basically, about both the technical and design challenges that need to be overcome.

[tra]

Could the manifest and ledi be combined? It seems like they serve a similar purpose.

[assertivist]

Yeah, having the manifest generated with a script, it could gather up the LEDI and store it right alongside the general index of files no problem. But I think we do want level set specific LEDI files so they're easier to edit for the designers and also hash for versioning. As long as set designers don't have to mess with manifest.json themselves.

[rherriman]

@tra I actually had a similar thought a night or two ago, but forgot to talk about it! I do like the idea a lot. I believe LEDI may even stand for "level directory" which is why I suggested the file could be named directory.json (honestly, I don't care what it's called).

I still dislike that the level designer would need to maintain the file whenever they add a new resource, some stuff in particular would feel egregious (like when adding a TEXT 1000 resource to set global variable names for referencing other resources)... but again, tooling can help here. I'm envisioning a command line app for "initializing" new level sets with a basic directory structure, adding new levels to it, and using file system introspection to auto-generate/refresh the manifesty bits.

It also occurs to me that maybe the text resources can also just be shoved into the manifest file directly. I'd need to do more research to determine what other "magic" text resource IDs may exist or how Avara loads them.

@assertivist I think we're gonna need to do more than hash the manifest for versioning, since any of the resources (particularly the SVGs) may have changed.

[assertivist]

Yeah TEXT and HULL can also just get shoved in there really. You might have to manually populate a struct or so for HULLs.

I agree, I think that each file should be hashed individually within the structure, including ledi.json (basically i was saying that the manifest shouldn't replace ledi.json even if manifest.json has a copy of that data in there)

[tra]

...using file system introspection to auto-generate/refresh the manifesty bits

I really like this idea. In general I'm not a fan of redundant information. Let the existence of the level files drive what is shown in the levelset list. But you might need a level naming convention to drive ordering. For example, order alphabetically unless the levels follow some naming conventions with numbers?

[assertivist]

Ordering (which we could also consider Searching) and display of level sets is definitely a concern when we're talking about converting all of Avara's user content, but I think that's firmly in the category of waiting until we have an ultimate level distribution system--think instead of ordering so that your levels are grouped, having meta tags attached so people can search them out by typing "gzr" for example

[rherriman]

Had this sitting around for a bit and wanted to post it before I forgot again. Here's what I was thinking for a "set manifest" file. (Not sure what to actually name it.)

{
    "id": "71c24c47-43eb-4276-8cc9-8981728ef0a8",
    "levels": [
        {
            "id": "6206b399-f736-4cdd-b30b-df9a4fef44a8",
            "name": "Fosfori",
            "description": "\rGlow.",
            "filename": "1000.svg"
        },
        {
            "id": "b2245e7e-49f2-4e28-b83c-0a1d33778509",
            "name": "Harha",
            "description": "\rFake.",
            "filename": "1001.svg"
        },
        {
            "id": "f77fc617-c406-4a4a-87ab-0b34cf45f465",
            "name": "Laine",
            "description": "\rInspiring.",
            "filename": "1005.svg"
        },
        {
            "id": "2cc3215d-90f1-4685-8471-64746f0ead82",
            "name": "Lammikko",
            "description": "\rFilthy.",
            "filename": "1002.svg"
        },
        {
            "id": "9b33f673-d4bf-4262-a614-d9f2c4790527",
            "name": "Silta",
            "description": "\rConnection.",
            "filename": "1003.svg"
        },
        {
            "id": "a3101140-ca1b-4254-8eb9-d9fa40c5835c",
            "name": "Sinappi",
            "description": "\rOrganic.",
            "filename": "1006.svg"
        },
        {
            "id": "3a7fb9e9-c32d-4a5d-b8f1-cc2ab8be71dd",
            "name": "Sinappi - Short",
            "description": "\rNatural.",
            "filename": "1007.svg"
        },
        {
            "id": "a1ceea51-2d41-404e-aae5-5fb222e500b7",
            "name": "Suuttumus",
            "description": "\rRash.",
            "filename": "1004.svg"
        }
    ],
    "script": "designer = \"silverfox <ryan@avaraline.ml.org>\"\r\rrFosfori = \"\r\rBright.\"\r\rrHarha = \"\r\rChilling.\"\r\rrLaine = \"\r\rRefreshing.\"\r\rrLammikko = \"\r\rPolluted.\"\r\rrSilta = \"\r\rJoined.\"\r\rrSinappi = \"\rKOTH.\rTeams of Green, Yellow, Red, and Blue.\"\r\rrSinappiShort = \"\rKOTH.\rSame level, half the time.\"\r\rrSuuttumus = \"\r\rAnnoying.\"\r\rsnBennett = 1000\rsnDavid = 1001\rsnJordan = 1002\rsnRett = 1003\rsnRyan = 1004\rsnSteven = 1005\rsnYes = 1006"
}

Note the usage of UUIDs and lack of enables/reserved attributes for levels (which could theoretically still be added later without breaking compatibility, as they could just default to 0). I ultimately decided to put the level IDs inside the level object rather than use them as property names (or hash set keys) in order to maintain the order of levels in the set. It will slow down searching for levels within a set slightly, which maybe would be a problem if a level set had thousands of levels in it. They don't.

Looking at this, I'm reminded that during the conversion process we should probably convert \r to \n as well as transcode MacRoman to UTF-8. I'd also suggest trimming level descriptions, which in this example would get rid of the leading \r. (I had put them in there to vertically center the text in the old UI.)

The script attribute is a replacement for the TEXT 1000 resource, but given all the quotation marks and line breaks in such a resource, it is probably better to move this attribute out of the manifest and store it in a separate plain text file in the same directory. I'd suggest a name like defaults.avarascript. A theoretical command line app would automatically create this file as well as a manifest when initializing a new set. (Avara itself could always try to load this file, and catch the exception if it doesn't exist.)

Still unsure on how I'd like hulls to be stored.

[assertivist]

Considering your comments and also looking at the output of the conversion tool, I've revised the interim structure a little bit:

levels/
├── manifest.json <-- contains a map of set tags -> folder name
├── set_name/ 
│   ├── set.json <-- contains LEDI, HULL, and sound metadata
│   ├── default.avarascript <-- TEXT
│   ├── bsps/*.json
│   ├── svg/*.svg
│   └── snd/*.ogg
└── ...

Putting all the data from all sets into one file looked cool, but even with the 36 sets we have it ended up being like 7000 lines. So the manifest.json will just be in the root to look up set folders until we have a better end-to-end indexing solution. Then the set.json will have the specific data like what you show in your example.

Hull data can be included in set.json. Here's an example from Balledness of what is exported now, it's pretty simple, just a few numbers:

   "150": {
    "Hull Res ID": 215,
    "Max Missiles": 8,
    "Max Grenades": 12,
    "Max boosters": 6,
    "Mass": 124.35913633936065,
    "Max Energy": 9.155275806820782,
    "Energy Charge": 4.5776455329213395,
    "Max Shields": 1.5258869306477454,
    "Shield Charge": 1.5258869306477454,
    "Min Shot": 1.1444113832303349,
    "Max Shot": 2.2888227664606697,
    "Shot Charge": 1.1444113832303349,
    "Riding Height": 0.3814755474174105,
    "Acceleration": 1.0528572518501564,
    "Jump Power": 0.9918364232852674
   },

[tra]

My first comment/question is whether the filenames would include "levels" in the path? For example
"filename": "levels/1004.svg"
Or is it presumed that they are (or are not) in the levels sub-directory?

Second question, can you accommodate multiple designers in a set? For example, the gzr sets typically have multiple designers. Should that be allowed to be overridden on a level by level basis?

[assertivist]

My first comment/question is whether the filenames would include "levels" in the path? For example
"filename": "levels/1004.svg"
Or is it presumed that they are (or are not) in the levels sub-directory?

Yeah it would just have it check for the file in levels/that_set/svg/<filename>. By the time it's looking for a level it already knows the set. Checking for a file is easy

Second question, can you accommodate multiple designers in a set? For example, the gzr sets typically have multiple designers. Should that be allowed to be overridden on a level by level basis?

That is already the case, the example is using the TEXT resource so that those script assignments would apply to every level (since he's the designer of all of them). Typically you would put designer = ... in a text block in the level drawing itself.

[tra]

Some other random thoughts regarding levelset formats

1. Several levelsets came with a PICT 129 resource which was a "splash" page for that levelset (e.g. gzr "Over the Hill"). Do we want to accommodate that? I don't even recall how these were displayed, maybe when you switch to the levelset? It might be that this just replaced the default splash page for Avara.

2. Likewise, some levelsets had a README file packaged with them. That wasn't in the levelset itself but often packed alongside the levelset and might have some additional details about the levelset. But it could also act a "credits" and "thanks" file more than anything. Would such things go in the manifest or somewhere else?

3. We might end up with 100s of levels. Do we want to consider another level of organization such as "guild" (e.g. "aa", "gzr") and have levelsets under that? This thought occurred to me in regards to the "gzr" levelsets because they are in alphabetical order instead of release order. I think it would be nice to somehow show the release order/date for a levelset. But maybe that could be another way of sorting levels at some point in the future.

[assertivist]

1. Several levelsets came with a PICT 129 resource which was a "splash" page for that levelset (e.g. gzr "Over the Hill"). Do we want to accommodate that? I don't even recall how these were displayed, maybe when you switch to the levelset? It might be that this just replaced the default splash page for Avara.

That image showed up in the space where the level descriptions did, but only if you didn't have a level loaded, IIRC. I'll check and take a screenshot, it could be a feature of the roster menu

2. Likewise, some levelsets had a README file packaged with them. That wasn't in the levelset itself but often packed alongside the levelset and might have some additional details about the levelset. But it could also act a "credits" and "thanks" file more than anything. Would such things go in the manifest or somewhere else?

We could easily support a credits line... Later on I imagine having a better interface for browsing levels. Data like that could easily be shown there.

3. We might end up with 100s of levels. Do we want to consider another level of organization such as "guild" (e.g. "aa", "gzr") and have levelsets under that? This thought occurred to me in regards to the "gzr" levelsets because they are in alphabetical order instead of release order. I think it would be nice to somehow show the release order/date for a levelset. But maybe that could be another way of sorting levels at some point in the future.

I wrote several paragraphs, and then decided that what you should do is go ahead and open an enhancement issue for "a better interface for browsing and loading levels". I know we've had some lengthy discussions in the past about several different ways to make that happen and support searching and tagging like you describe.

[rherriman]

1. Several levelsets came with a PICT 129 resource which was a "splash" page for that levelset (e.g. gzr "Over the Hill").  Do we want to accommodate that?  I don't even recall how these were displayed, maybe when you switch to the levelset?  It might be that this just replaced the default splash page for Avara.

It appeared in place of the Ambrosia SW logo, under very specific conditions. (Freshly opened Avara, opened the level set, did not load a level, cover up the logo with another window, then uncover it.) Personally, I was not planning on supporting this feature.

Regarding the README and set tagging, I think this is dependent on how levels will ultimately be distributed and installed. If people are manually downloading and installing asset bundles, a README should probably exist outside the bundle, for example.

Secondary organization of sets could be handled from within the "manifest" (a term that is becoming increasingly inaccurate, I think), or possibly as metadata on a centralized asset server. As an example here, if we could validate that a level set belonged to a certain author/guild (because it was uploaded by a particular account), that would be better than trusting whatever the author put in their manifest file.