Space Exploration compatibility

[Genhis]

Hi Deadlock!

First of all, you have created an excellent mod. I've been enjoying both IR and IR2 - the slow-paced progression, your graphic style... When I tried Space Exploration, I felt that I am progressing too quickly early on and I didn't like some recipe decisions of AAI Industries, namely the fact that I needed to supply the previous tier machine to create a new one.

I remember that you considered making your mod compatible with SE in the past. Would there be something I could help you with, to make IR2 compatible with SE? (Lua coding, recipe conflicts / balancing, technologies, etc.)

Once again, thank you for your work, it's really appreciated!

[Deadlock989]

You're welcome.

Realistically, this is not something I am ever going to have time for. I still haven't played SE, and I have less gaming time available to me now than I did in previous years (and I'm not spending much of that time on Factorio).

There are some obstacles that you will run into before you even start looking at how to blend IR2's tech progression into SE itself:

1. AAI Industry is compulsory to play SE and it's extremely difficult to work with. IR1 had some compatibility for AAII - it was very difficult to sort out. Linver, the main coder for Krastorio, put a lot of time into making K1 work well with IR1 - but making IR1+K1+AAII all load at once defeated us both, so we gave up. When it came to IR2 and having to do that all over again, I decided life is too short.

2. Alien Biomes, another hard requirement for SE, disables IR2's rubber trees semi-randomly depending on load order (which depends on which other mods you have installed and even whether you're on Windows or Linux). Dependencies can fix that (but might also constrain you in other ways).

3. SE currently has IR2 marked as incompatible. People would have to manually remove that from SE's info.json after every SE update.

If it's something you want to have a go at, you could try making a "patch mod" that has both IR2 and SE as dependencies, or maybe SE's "post-processing" mod. IR2 makes a lot of functions public in a big table of functions (DIR), some of it is even vaguely documented. See RusselRaZe's compatibility patch mod for an example of someone using them (a bit). IR2 is essentially "done" and not ever going to change very much, so you can assume that's it's reasonably stable. If I do ever get the itch to expand it, it will be through separate add-on mods.

[Genhis]

That's very helpful, thank you!

I have managed to get IR2 running with both AAI and SE. There are some minor issues in the tech tree (it needs a bit of reordering) but it shouldn't be hard to fix. Balancing everything may take longer but again should be doable.

However, I ran into one major issue: DIR.recursive_add_pack_to_tech assumption about no cycles doesn't hold for SE. To get it working, I had to add this if statement directly to IR2:

if string.sub(tech_name,1,string.len("se-"))=="se-" then return end

I am wondering, is there a better solution?

[Deadlock989]

That sounds odd, shouldn't happen.

IR2 builds its tech tree dynamically. IR1 did this for all techs it could find, which caused a lot of issues with other mods. IR2 now only does this for its own native techs and also for a hardcoded list of all the techs that vanilla provides as of Factorio 1.1. You can force the tree builder to treat any other mod's technology as "native" (and automatically slot it into the right place in the tree, based on the ingredients of its recipe unlocks) by adding a property IR_native = true to the technology prototype. (This isn't intended to be done for "enriched" or "advanced" technologies which unlocks additional recipes for items which have already been unlocked earlier in the tree, e.g. Coal Liquefaction, Kovarex Enrichment.)

If the tree doesn't have any cycles in it when the game gets past loading all the mod prototypes, but it does have cycles after the tree builder has run but before the recosting ... I'm not sure how it could get into such a state. Possibly something about the SE techs is being changed at a later stage (maybe in data-final-fixes) which allows the game to load, but before then the tree is in a cyclical state?

You could try adding the IR_native property to the problematic SE techs and see whether the tree builder can resolve the cycles on its own, but at this point I would probably be starting to dump technology prototypes to the log to find out which ones are involved, and then looking in the SE code to see where they are being set up. The main thing is that the tree should be sane before IR2 hits its data-updates stage.

[Genhis]

While I can't be 100% certain that there are no cycles, I believe so. I think the issue is that SE has a huge tech tree, so when you traverse children of technologies, you will traverse the same ones many times. This issue exists in "vanilla" IR2, only IR2 doesn't care about traversing items multiple times because the tree is small enough to finish in time.

For example, let's take the bronze analysis pack and "Electronics 1". It gets added the pack and its children get processed. "Electronics 1" has a child "Inserters 1", so that will also get the pack. However, "Electric motor" also has "Inserters 1" as its child, so when "Electric motor" children are processed, "Inserters 1" and all of its children get bronze analysis pack added for a second time.

A more extreme example:

Concrete + Optics + Improved laboratories 1 = Iron analysis

Everything after iron analysis gets processed at least 3 times (likely even more) because for each point you are traversing all nested children of any technology. So, a late-game SE technology may get processed thousands of times.

To fix this, you could use a table to store processed values and stop a branch if you find that it has already been processed.

function DIR.recursive_add_pack_to_tech(pack, tech_name, count, processed)
	if not processed then processed = {} end
	if DIR.list_contains(processed, tech_name) then return end
	table.insert(processed, tech_name)
	-- Do the rest
end

With this fix, I was able to load the game and it even added "missing" science packs to SE techs.

What do you think?

[Deadlock989]

OK, when you said "assumption about no cycles doesn't hold for SE" I assumed you were talking about a cyclical tech tree (a tech tree with a looping structure of prerequisites).

Well spotted with the inefficient recursive search. I thought I already added caching here but now realise I only did it for the earlier tree structure search where it works out prerequisites.

I feel like this would be even faster (at some temporary memory cost):

function DIR.recursive_add_pack_to_tech(pack, tech_name, count)
	if DIR.recursive_pack_cache == nil then DIR.recursive_pack_cache = {} end
	if DIR.recursive_pack_cache[pack] == nil then DIR.recursive_pack_cache[pack] = {} end
	if DIR.recursive_pack_cache[pack][tech_name] then return end
	DIR.add_pack_to_tech(pack, tech_name)
	DIR.recursive_pack_cache[pack][tech_name] = true
	local children = DIR.find_techs_with_prereq(tech_name)
	for _,child in pairs(children) do
		DIR.recursive_add_pack_to_tech(pack, child)
	end
end

Please try this out and if it works I will include it in the next release.

[Genhis]

Yeah, I assumed there was a cycle but your cycle detection system would have caught that, so I guess there wasn't one.

IR2 alone works fine, technologies seem to have correct science pack progression.
IR2 + SE also works fine, apart from a few technologies which are overridden by SE at later stage.

I encountered another cyclic technology dependency in SE at data-final-fixes stage, but I should be able to fix that.

Thank you, hopefully no other fixes will be needed from your side. I will get to work now. :D

[EarendelGames]

I'll remove the IR2 incomaptibility line once things are working togheter well. It's best if players don't try to play the mods together too early otherwise changing things can make a big mess of people's save files. They can of course edit the info.json. Just be aware that SE compatibility is a moving target, expect SE to keep changing a lot over the next 2 years.

[EarendelGames]

I'm not sure what the problem with rubber trees is, but it sound like it might be easier to fix on my end?

[Deadlock989]

I'm not sure what the problem with rubber trees is, but it sound like it might be easier to fix on my end?

The problem is on lines 15-19 of alien-biomes/prototypes/entity/trees.lua - it deletes the autoplace from every tree prototype in the game at that point, regardless of whether those trees are from the vanilla base mod or whether they are a key resource from some other mod.

If AB loads before IR2 (which seems to be the most common case), there's no problem - IR2 creates a new tree (ir2-rubber-tree) and sets up its autoplace after AB has taken the action above. If AB loads after IR2 then AB disables IR2's new tree along with all the others. Rubber trees drop rubber wood; rubber wood is converted into rubber and more rubber trees; having no rubber wood stops any progression past the early electric phase of IR2.

I have had reports/complaints from Linux users who seem to think that Linux and Windows will load the same combination of mods in a different order because the former has a case-sensitive file system and the latter doesn't (upper-case I in IndustrialRevolution versus lower-case a in alien-biomes). I haven't established whether this is actually true or not, but certainly we have found that different players with similar mod combos can get a very different load order.

The simplest band-aid is to add an optional dependency on AB to IR2, or to defer IR2's resource generation to data-updates, but that feels like addressing the symptom rather than the cause, and for all I know, there might be some other case where people need IR2 to load before AB. In general I don't add dependencies unless I have also taken the time to add explicit support for those mods, e.g. AAI Warehousing - I don't even know if IR2 rubber trees spawn as intended with AB's temperature/humidity/elevation variation (one user has reported it was fine in one game).

[Deadlock989]

Thank you, hopefully no other fixes will be needed from your side. I will get to work now. :D

Best of luck with it. Closing this issue to avoid separate conversations on different topics, if you have other questions, open a new issue.

[rekales]

Another one into fray

Hey guy, I'm another modder who's gonna attempt to make a compatibility patch for IR3 and SE and I'm currently requesting for your permission to do so.
I'll be doing all the work and will just inquire suggestions and request api stuff when the need arises.

I would also have the compatibility code integrated into SE if possible as per suggestion by other modders. The SE guys seems cool with it.

As for progress, I recently started so it's not much.

[Deadlock989]

You don't need my permission. I have no objections.

For requests please create new issues.