Scenario not visible after unzipping (Variants no longer indexed)

[RogerCooper]

How can the problem be recreated?

Put the file in the downloaded maps folder. It unzips it, but the "5 if by Sea" scenario still does not appear when selecting

Do you have any ideas for an expected fix?

Probably some question of folder structure for a scenario that does not come packaged with its own map
five_if_by_sea.zip

Attach a Save Game

If playing a prerelease, which version are you using?

2.6.36
Game Version:

If playing a prerelease, does this happen on the latest release?

• [] yes
• [] no

Is there anything else we should know?

[DanVanAtta]

This looks to be expected behavior, support for map-variants was dropped IIRC in 1.8 or 1.7.

[RogerCooper]

These were all working fine until your recent changes. Why should it not be possible for an XML to reference the map from another folder? Isn't that the point of the mapName property? If you can't reference another map, all the files need to be copied over, resulting in needless duplication.

[DanVanAtta]

These were all working fine until your recent changes.

Those changes were three and a half years in the making and since then we've repaired all variants to have a full set of game files in preparation. You can read more background on the discussion and reasoning here: #1247

Isn't that the point of the mapName property?

It is deprecated and no longer used as of 2.6, a long time in the making. The 'map_name' property and the way maps were found had a number of design flaws that made it unreliable, complex, and brittle.

If you can't reference another map, all the files need to be copied over, resulting in needless duplication.

For now at least, until XML files are completely decoupled from maps, yes.

[DanVanAtta]

For context, there is already a very high level of duplication between maps. Duplicated content is not a new problem, but it's also not the worst of problems either. Variants present a number of additional problems.

I think you have two realistic solutions:

1. copy original map content into the variant
2. move the variant XML file to the required map

I can imagine ways where we could create some hacks to the map.yml structure such that variants would still work. I'm not going to pursue those. I believe the correct long term solution is to decouple XML files from map content.

[DanVanAtta]

This is an interesting example:

~/triplea/downloadedMaps/NekahNets_1939_3$ ls
baseTiles  centers.txt  games  map.properties  map.yml  place.txt  polygons.txt  smallMap.jpeg

This map is not one controlled by TripleA so we are unable to update it, and even though it is perhaps a decade old at this point we still have to actively deal and code around it. This variant is interesting:

1. All the files in it are useless apart from the XML. I am surprised.
2. The map name is 'big_world', even though it says it is based on pact of steel
3. the map name attribute is being used to generate to 'map.yml' files where missing to avoid players from having to re-download all maps. This causes the 'map name' to be wrong and when looking for 'big_world' the variant is retrieved instead of actual 'big_world'. I'm wondering currently how to solve this and it is posing an active problem and has cost a pretty decent bit of time today already.

[RogerCooper]

The link #1247 was from 4 years ago and I didn't see anyone other than you saying it was a good idea. (Cernelius was taking the same position I do). It simplifies the code for loading but that code was already working and did not need to be changed. A GitHub case is not a good place to discuss major changes that deprecate existing mods. This should have be discussed in the Map Engine Development section of the forum. Deprecating an important existing feature like mapName needs more discussion.

As a map-maker I find that TripleA does not have enough dependencies. For example, if I want to add Australian units into a mod I can't just reference the World War II Global game, but instead need to copy everything. That was before you made your changes.

I like the idea of a separate graphics pack, which could be referred to using the mapName property. I suggest reverting to the older system of handling files until your are ready to actually use graphics packs.

[DanVanAtta]

The link #1247 was from 4 years ago

Exactly, this was decided quite some time ago. We updated all existing known variants after that decision.

This forum post has been open for 4 months to detail the final steps being taken to deprecate 'mapName' https://forums.triplea-game.org/topic/2333/proposing-maps-yml-index-file-to-be-added-to-each-map

[DanVanAtta]

It simplifies the code for loading but that code was already working and did not need to be changed.

What's your evidence to support this opinion @RogerCooper ?

[DanVanAtta]

I suggest reverting to the older system of handling files until your are ready to actually use graphics packs.

No, it's too much, too far baked, it's been up for discussion for half a decade now regarding variants and almost half a year now (minus 2 months) for deprecating mapName. In addition, it is the lack of variant capability that seemingly your objection. The migration of mapName is tangential and has further purposes (notably to remove triplea_maps.yaml).

The migration of map-name is only related because those brittle designs make that migration more challenging. Those challenges are evidence of why things needed to change, there are other efforts that have been impeded by overly complex code.

[DanVanAtta]

The link #1247 was from 4 years ago and I didn't see anyone other than you saying it was a good idea

@RogerCooper, Ron wanted to go further and disallow more than one XML per map. An even more extreme view than what I had proposed.

Note this comment: #1247 (comment)

Also note this comment that is in agreement: #1247 (comment)

[DanVanAtta]

@RogerCooper this comment describes several benefits, it was not just for simpler code:
#1247 (comment)

The complexity and the knots which the TripleA codebase is tied into represent an existential threat to TripleA.

[DanVanAtta]

Summarizing, @RogerCooper I don't think you fairly summarized the conversation from #1247. The removal of variant support is an old decision at this juncture. I was unaware that there were still un-official variants that were expected to work. What would the trouble be to simply update those variants to have their assets? There are a tremendous number of assets duplicated between maps already, this is really not a new problem. To get to map graphic packs, we need a maps server. It will be many steps to get there, if we ever do.

[Cernelius]

@DanVanAtta As I've most likely said multiple times, I believe you are making it harder for game-makers to play-test game-only modifications of existent maps before releasing any one of the modifications. This can be currently done by having a game file not necessarily within the referring map. Of course, one can just clone the entire map changing the folder name and then, once the new game is completed, paste the game file inside the intended map in substitution of the present and now obsolete one and then proceed to update the repository accordingly, but this means that it is not meanwhile possible to test the modification but with persons which manually install it. Of course, play-testing can be guaranteed simply by keeping all game files of the same map within the game folder of the map, but the main problems of keeping such an additional and temporary/experimental game file within the game folder of the map are that, this way, it is much harder to keep track of where all your game modifications are (currently, you can have them all within a single folder, instead of having to disperse them all across their maps) and you risk losing it anytime by re-downloading the map. Of course, this matters only for map-makers that playtest their maps in one or more modified game states in between of officially releasing versions and could be avoided by keeping updating the repository at every change you make, but I believe this is annoying to users if too frequently done.

---

For example, I have this directory:
triplea/downloadedMaps/+/games
where I keep all my unreleased (and possibly never to be released) game-only modifications of existing maps.

For example, now I have this game file:
triplea/downloadedMaps/+/games/270BC_Wars+.xml
which is waiting to be renamed to 270BC_Wars.xml and substituted within the "games" folder of the "270bc_wars" map once it is ready.

Immagine having tens of such game files dispersed across their maps instead of grouped within the same folder.

[RogerCooper]

As you noted, duplicated files are already a problem in TripleA, but that does not mean that it is OK to make the problem worse.

This sort of major change, breaking many existing mods, and deprecating a major feature, should have been discussed in the forum. Few people are subscribing to every case in GitHub.

You asked what evidence I have that this was working before. As you scraped the mods I had stored on my site, you can test them yourself. They all ran before you made the change that eliminated the zip files.

[DanVanAtta]

@Cernelius , thank you for the response, my responses are inlined below with some questions:

I believe you are making it harder for game-makers to play-test game-only modifications of existent maps before releasing any one of the modifications. This can be currently done by having a game file not necessarily within the referring map. This can be currently done by having a game file not necessarily within the referring map.

If you are doing a variant to achieve this, you are still required to modify the original map. The only difference is where the XML lives. Is there some other way that variants helps? Is it that hard to track that you have a new XML file plopped into an existing map? If you are modifying existing maps, aren't you already taking a copy of those maps to leave the originals intact? 'Git' is designed exactly for tracking file changes, it is perfectly suited for this very task and could be used to help this.

Of course, one can just clone the entire map changing the folder name and then, once the new game is completed, paste the game file inside the intended map in substitution of the present and now obsolete one and then proceed to update the repository accordingly, but this means that it is not meanwhile possible to test the modification but with persons which manually install it.

This requires a manual install either way, no? If planning to distribute the variant, either the XML needs to be packaged back into a map one way or another. If what you're saying is that it's easier to keep a completed map intact and unmodified, but then distribute folder zips that contain a single XML, I can understand that. At the same time, it should be readibly possibly to bundle the needed assets into that zip as well.

Of course, this matters only for map-makers that playtest their maps in one or more modified game states in between of officially releasing versions and could be avoided by keeping updating the repository at every change you make, but I believe this is annoying to users if too frequently done.

Wouldn't multiple XML files for the differing game states satisfy this? Couldn't you also launch a 'variant-beta' version of an existing map for this as well that would be frequently changed? Alternatively the in-work XMLs could land in the main map as well, it is only possibly annoying to users if there is a request to update from an out-of-date version. There is no requirement to update the out-of-date notification until you want to.

[DanVanAtta]

@RogerCooper

As you noted, duplicated files are already a problem in TripleA, but that does not mean that it is OK to make the problem worse.

Variants create other problems. One signficant problem is compatibility and unknown dependencies. There are further problems created by having 'map-name' not actually refer to the map holding a XML and assets. The variant scenario I pointed to is an example of this where it was incorrectly created and all the assets bundled with that variant are needless. I am stating that the additional complexity is a major problem and not every map maker is able to get 'variants' "right". Last, variants is just not a good solution to the duplication problem, it was never really intended to be. It was a way to re-use existing maps without having to involve the original map maker to reequest they add an additional XML to their distribution.

This sort of major change, breaking many existing mods, and deprecating a major feature, should have been discussed in the forum. Few people are subscribing to every case in GitHub.

I vaguely recall this was mentioned in the forums. Back 4 years ago, it was rare for anyone other than developers to be involved in a github issue discussion. The fact that others chimed in, indicates there was a further outreach and a major decision to deprecate official support for 'mods' would have involved others (as it apparently did).

You asked what evidence I have that this was working before. As you scraped the mods I had stored on my site, you can test them yourself. They all ran before you made the change that eliminated the zip files.

I was not asking for evidence that this working before, but for evidence of why you stated that the code around this did not need to change. I'm curious what observations you have, or evidence, to know about that the 'map-loading code is just fine and is not getting in the way of anything else and can simply never change again. I apologize for asking a disengenious question and I should have shown more courtesy.

Analyzing the mods I obtained from 'roger-cooper.com, this is the list of mods

270BC_Tertiered
270BC_Variants
AAC-BuildCaps
NE_Variants
NWO-Finest_Hour_The_Tipping_point
PearlHarbor1941
WorldWarIIEmptyStart
ww2global40_ryuzaki
ww2global42_2nd_edition
WW2v5_1942_2nd_Edition_High_Production_Mod

This is frankly not an insurmountable list to fix. Each one of the mods listed above contain a single XML file. Is it possible to update these on roger-cooper.com? In addition or alternatively can we move these to official support? Are some of these mods experimental and could be dropped? Are any of these mods in a complete enough state where we could simply add them to the core map?

[DanVanAtta]

Summarizing, the crux of this thread is that while 'variants' lost official support 4 years ago, the mechanism was still being used despite lacking official support. To this extent I think we should work together for everyone to get on the same page to all use the newest and best way to modify maps.

This leaves in limbo the dozen unofficial map variants that are going to be in newly broken state now that the mechanism to load map variants is changed. Presumably these are all upgraded to 1.9 compatibility, otherwise if they were 1.8 maps then they would have been broken for the last 2 or 3 years. This implies an ability to modify these distributions, if so we have some options there.

We could debate whether we want to bring back official support for map variants. This breaks the design that maps be fully self contained. There are further implications as well.. In short, this is an absolute last resort. Considering the impact is that a dozen unofficial XML files would need to be dropped into existing map installs, and they are being installed manually already, I'm saying it's a no and strict last resort to reversing a decision that was made by the group 4 years ago. This leaves in question how to best fix the impacted variants that we are now aware of.

[DanVanAtta]

@Cernelius I just realized, with map.yml you can specify the path to individual game-files. You could easily create a folders like these:

/unofficial
/work-in-progress
/unreleased
/experimental-variants

Seemingly that would keep the work-in-progress items cleanly delineated. As of 2.5, you can create that folder structure today, the engine parser will look for all available XML files and try to parse them. Since 1.9, you can extract a map zip and it should be readily available in the engine, modifying the original map is certainly therefore possible (so you don't have an issue where you have to unzip and re-zip constantly to play-test mods).

[Cernelius]

@Cernelius , thank you for the response, my responses are inlined below with some questions:

I believe you are making it harder for game-makers to play-test game-only modifications of existent maps before releasing any one of the modifications. This can be currently done by having a game file not necessarily within the referring map. This can be currently done by having a game file not necessarily within the referring map.

If you are doing a variant to achieve this, you are still required to modify the original map. The only difference is where the XML lives. Is there some other way that variants helps? Is it that hard to track that you have a new XML file plopped into an existing map? If you are modifying existing maps, aren't you already taking a copy of those maps to leave the originals intact? 'Git' is designed exactly for tracking file changes, it is perfectly suited for this very task and could be used to help this.

I don't need to make any copy of a map in the moment I'm changing nothing in the map and just creating a game file that uses the map. As I said, it is much easier to have all game-only modifications grouped within the same folder. Besides, there is also the risk that I put a game-only modification inside a map and I forget about it and overwrite the map, losing the file. However, this is really my personal practice, as I suppose others rather have one "variant" folder for all the game-only modifications of one map. I don't see a point in that, since I can name the game file itself to be clear enough what it may be for without having to open it. I don't want to keep using Git or rather use it at all but only for when I make an official release.

Of course, one can just clone the entire map changing the folder name and then, once the new game is completed, paste the game file inside the intended map in substitution of the present and now obsolete one and then proceed to update the repository accordingly, but this means that it is not meanwhile possible to test the modification but with persons which manually install it.

This requires a manual install either way, no? If planning to distribute the variant, either the XML needs to be packaged back into a map one way or another. If what you're saying is that it's easier to keep a completed map intact and unmodified, but then distribute folder zips that contain a single XML, I can understand that. At the same time, it should be readibly possibly to bundle the needed assets into that zip as well.

Nothing is being distributed, so no installation, manual or not, is required. Just you have this unreleased game file (using the skin of a released map) which you can play with everyone having the map, as long as you are hosting. I'm not plannig to distribute any skin-less maps. By "unreleased" I'm not implying that there is any intention of releasing it.

Of course, this matters only for map-makers that playtest their maps in one or more modified game states in between of officially releasing versions and could be avoided by keeping updating the repository at every change you make, but I believe this is annoying to users if too frequently done.

Wouldn't multiple XML files for the differing game states satisfy this? Couldn't you also launch a 'variant-beta' version of an existing map for this as well that would be frequently changed? Alternatively the in-work XMLs could land in the main map as well, it is only possibly annoying to users if there is a request to update from an out-of-date version. There is no requirement to update the out-of-date notification until you want to.

Here we are talking about files I may be keep changing and testing once every few minutes overtime. Besides, I don't like releasing versions until I feel they have a chance to be at some definitive state. For me Git is only for releases, not for pre-releases.

[Cernelius]

@Cernelius I just realized, with map.yml you can specify the path to individual game-files. You could easily create a folders like these:

/unofficial
/work-in-progress
/unreleased
/experimental-variants

Seemingly that would keep the work-in-progress items cleanly delineated. As of 2.5, you can create that folder structure today, the engine parser will look for all available XML files and try to parse them. Since 1.9, you can extract a map zip and it should be readily available in the engine, modifying the original map is certainly therefore possible (so you don't have an issue where you have to unzip and re-zip constantly to play-test mods).

I don't understand the matter clearly enough.

[DanVanAtta]

@Cernelius put simply, if all you want is a distinct folder for work-in-progress XMLs, you can create that today within a map. The (2.5) engine will look for XMLs in any location, you can create whatever sub-folders you wish beyond the 'games' folder. In 2.6 this will change and the engine will look at any location that the 'map.yml' tells it to.

[DanVanAtta]

And yes, in 2.6, the path in the map.yml file can be ../my-variant-folder/variant.xml, and I would expect that to work just as a variant has previously. To get that to work, you'll need to modify the 'map.yml' file of the original map.

Whether the variant folder is a sibling folder to the map, or a sub-folder, in either case you need to modify the original maps 'map.yml' file. Given that, I would think it would be just as well to create a sub-folder in the map for variants.

[Cernelius]

@Cernelius put simply, if all you want is a distinct folder for work-in-progress XMLs, you can create that today within a map. The (2.5) engine will look for XMLs in any location, you can create whatever sub-folders you wish beyond the 'games' folder. In 2.6 this will change and the engine will look at any location that the 'map.yml' tells it to.

All I want is having a bunch of game files which are all inside a single skin-less folder and each of which uses a map as its original skin (without having to add the game file inside the map it uses because I prefer keeping the map as it is given to everyone in download list).

[DanVanAtta]

Custom host-only XML support cannot be maintained long term.

In 2.5 you can create a sub-folder to for those XMLs. Simply don't check in that folder for official upload. In 2.6 you can hack the map.yml to get a folder in a different directory, but you'll then need to be careful to not check in the map.yml file. In any case you are able to preview all changes to an official release, presumably this is something you do so you know what you are changing exactly and you'll have an opportunity to abort and remove unwanted changes.

[RogerCooper]

There a lot more mods to be fixed than on on your list (including some good ones). The problem isn't fixing them, it is that xml-only mods must be changed to full mods.

(It was harder when I had to unzip every file, fix the code and then zip it up again).