Lua file scraper (for mods) #72
Comments
|
I recently forked a calculator from KirkMcDonald, the goal was to rewrite all the code on Go in pure Lua. Result — https://bitbucket.org/hhrhhr/factorio-tools-lua Since Lua 5.3 was used, some mods that actively use modified Lua 5.2 cannot be loaded yet (for example, angelsrefining_0.11.10). However, the rest of the mods are imported without problems, more precisely, they work the same way as in the original factorio-web-calc. The developers shared the source code of modified Lua, so I can update this repository. I can also try to provide data.raw and a file with icons in the format you need. |
|
Here is the test result from vanilla. Obtained completely automatically from game files, the code for using most mods is almost ready. There are some rough edges (like displaying empty / full barrels), but all the data and values are correct. In the near future I will publish the source code of the parser, but maybe there will already be some comments? For my part, I want to note that the calculator really needs the ability to load external data (textures with sprites, a css file and, of course, a data file).
|
|
P. S. Аnd here I tried to "render" how sprites will be arranged using the sort order from the game.
|
|
So, I represent a completely (almost) workable parser - https://bitbucket.org/hhrhhr/factorio-lab-tools The downloads have a ready-made set of all the libraries in the archive (unfortunately, only for Windows x64). In fact, GNU/Linux requires Lua, lua-zzip, lua-gd, and several other modules in pure Lua. After starting the utility, the following files will appear in the current directory:
There are several difficulties. I don’t know yet how to use the available tools to “color” the icons of the barrels of liquids (now they are all the same). I also had to forcibly remove the "burner-mining-drill", since it consumes coal, which it itself produces, which leads to a loop and an error. I don’t know yet, is this the result of incorrect export or a feature of Factorio Lab :) I enclose ready-made files from 0.18.26. if everything is fine with them, it remains only to connect the processing of mods (in archives and unpacked). |
|
Thanks for your effort! I haven't had the time to work on this very much in the past few days, I'll check it out asap. |
|
Btw, if anyone is having trouble dealing with some of the more complicated mods feel free to reach out to me. I helped update Foreman to support the larger mods like Bobs. There a few things that are hard to anticipate... Foreman: https://github.com/Rybadour/Foreman |
|
So I've been kicking this idea in my head lately... Since there are now a few different tools that use item and recipe data from mods. We should try to focus all our attention on solidifying this scaper tool. One addition to that is instead of us parsing mods at runtime we could also ask mod owners to run this tool to generate a JSON file they publish along side the mod. |
|
@Rybadour, now you have me imagining a scenario where mods could even build-in an SVG representation of the icons as part of the JSON itself, rendering the sprite sheets unnecessary. And if there were a hosted repository of these mod JSON files, apps like Factorio Lab could run against those at runtime rather than requiring them to be built in. Imagine if mods.factorio.com had a public API that could return the types of items, recipes, and icons that the mods added. |
|
Yes! Maybe we can ask the Factorio devs for this. It would save us a ton of effort... |
|
64px version from 0.18.27 ;) |
Some mods have their own settings. For example, you can choose to use water in recipes or change the color of the ore to blue. In this case, you need to ask the authors of the mods to generate all possible variants of their configurations, which will lead to a power-law increase in the number of these configurations depending on the number of settings... Therefore, I believe that "parsing mods at runtime" is indispensable. |
|
I mean if some items are conditional then that could be incorporated into the tool that generates these settings. Another mechanism could be to generate the 99% that are static and a separate file that describes the configurable 1% or the describes all that encompasses the 1% variants. |
|
@hhrhhr I think you are most likely correct, but obviously it'd be much nicer if we could handle mods at runtime instead of build time. In my opinion the best case is a hybrid where you have to use a tool like yours to generate files for your mod set, then you can load those into the app at runtime instead of build time. That's what I plan to work towards for now. |
|
Big update. Added parser support for mods (so far only unpacked), icon painting (using tint), support for arbitrary scaling (48px, for example), the ability to disable mod loading or texture generation. Examples (as on the top image) — https://hhrhhr.bitbucket.io/ Issue: tthe technology tree is not scanned; all possible recipes and products are displayed. Sorry, but the endless technology and the new json file format are still at work ;) But the resulting data.raw almost perfectly matches what can be pulled out by the mod when loading the game. I saw the difference only in the 8-10th digit of numbers with double precision. |
|
Current planned strategy for mod support:
Thoughts? |
|
Sounds like you don't plan to read mods dynamically? Foreman has had great success reading the mods directly from the mod directory. The biggest mods only take several seconds to parse everything. |
|
I would love to but I don't see how it could be done. The browser doesn't have permission to directly read files from the local machine, and doesn't have lua language support. If you know of a way then I agree it would be far better to read dynamically. |
|
I think you could load the json file via a request and then consider a setting that lets you provide a new json file. The json file would need to change slightly to contain a url to the sprite file but otherwise it should be able to load everything at runtime. Obviously all the compile time dependencies to the json file would need to be removed (certainly non-trivial)... Parsing the mods directly out of the mods directory is probably a step too far. I think that might actually be possible with file apis though some day? For now it is probably good enough to expect the mod data is available at a http get url which allows basic unauthenticated CORS requests with no additional headers (raw static json/image files at public urls). |
|
Where would this mod data be hosted? As far as I know no mod data like this currently exists. If there were a public REST API for mods that'd be simple but I think we have to start at square one. |
|
Github pages seems like a good enough place to me to host these files. Some of them could even be static assets in the https://factoriolab.github.io site published as part of the build. Anybody who wants to load a mod could put a repo with a json file and image up and then put the url to the json file in a setting. And you are correct it doesn't exist right now. |
|
I think we are largely on the same track, then. The plan I outlined just exclusively has the files hosted inside factoriolab.github.io. In order to read from other sources either Factorio Lab would need to recognize those sources or there would need to be a way to specify external sources in the UI, which is feasible, just doesn't seem as user friendly. |
|
Ah sorry I misunderstood. I thought we were talking about a script the user runs on their computer to generate mod data as a JSON file. I guess that's one advantage Foreman has if we want to parse mods we simply find the mod directory and extract some zip files. So FYI, working with Foreman has taught me mods are very complicated. For example, many of their Lua only work when they can import other mods they depend on. Some imports modules from core or base so those would have to be available as well. |
To be fair, yes, that is what I'm talking about. However, I don't intend that every user has to do that, only if you are using a mod that is not yet supported. In that case the user would run a script, generate the files, and create a PR to merge them into the repository, and after that the next person that wants to use that mod should be able to use the built in files. Using generated JSON at runtime should help us avoid Lua confusion. If we can generate the files, then we have a manual step to make sure they're valid, then they have to go through a PR before they're available for anyone to use, then the tool to generate the data files doesn't have to be 100% perfect. Dependencies will be complicated to handle, if a recipe from one mod requires ingredients from another mod it will either have to automatically enable the other mod, or will have to be handled like wood where there is no recognized recipe for the item until the dependency mod is enabled as well. |
|
If the sources and compiler are exactly the same, then the difference can only be in the system dynamic library msvcrt.dll. Although I also have no idea what exactly this difference could be :(( |
|
I'm not 100% sure what I did differently, but I got it working as expected this morning. A few things I cleaned up: I had to re-add the lcc directory to my PATH (apparently it was missing), I was able to rebuild using mass_make_tcc without any local modifications (previously I'd had to tweak Minor update: It seemed bad to me that your app had to include the time and empty module icons just to get my app to work. I've refactored so that those icons are provided directly by the app no matter what (as well as all the other icons that are explicitly used, mostly in the settings panel). I'm going to try to see if I can build data for 0.16, 0.17, and 0.18 using your tool and include that in my latest branch. Main thing is that I want to get it to include the space-science-pack. |
|
After updating the def files for TCC, the hacks for floorf are no longer needed. It would be necessary to remove them altogether from the source, somehow I will do it. About previous versions of the game. I don't remember exactly, but they used slightly different calculations for the speed and volume of conveyors and something else.
It will be necessary to somehow determine the version of the game. |
That's true, I hadn't really considered that. I think that was true in 0.16 and hasn't been true since 0.17+ though, so a manual tweak to the numbers for belts in the 0.16 json shouldn't be a major issue, as long as it's not necessary to support a wide variety of 0.16 mods as well. |
|
@hhrhhr Thanks for all of your help so far, Factorio Lab now supports 0.16, 0.17, and 0.18 thanks to you! |
This was easy ) https://bitbucket.org/hhrhhr/factorio-lab-tools/commits/b2ab43da649ea6b19670bb0c8cdde9447221b0c9 However, the problem remained. I expected that mods will use a call like |
|
Krastorio is now exported seemingly without errors (apart from those that are present in the game's log). p.s. |
|
So I synced with your changes and I still don't see any mod data generated when I run the tool. I tried again with both Waterfill and Krastorio 2, in both cases the output is the base 0.18 game data with no mods. With Krastorio 2: C:\Git\factorio-lab-tools\lua_lab>..\dist53\bin\lua53 factorio_data_dump.lua --gamedir "%factorio%" --iconsize 64 --clear export
[INFO] mods settins loaded
[INFO] mods load order received
[INFO] load order applied
[INFO] core and data load...
[INFO] exporting data.raw...
[INFO] process all items... found 162 with type="item", 78 with other type
[INFO] process technologies... enabled 181 recipes
[INFO] process recipes... found 199 normal, 18 expensive, 15 hidden
[INFO] sort recipes...
[INFO] found 82 ingridients and 195 products
[INFO] clear items...
[INFO] icon's canvas 16x15 (|xx| show number of layers)...
row 1 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 2 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 3 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 4 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 5 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 6 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 7 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 4|
row 8 | 4| 4| 4| 4| 4| 4| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 9 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 10 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 11 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 12 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 13 | 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1| 1|
row 14 | 1| 1| 3| 3| 3| 3| 3| 3| 3| 1| 1| 1| 1| 1| 1| 1|
row 15 | 1| 1| 1| 1|
[INFO]
all doneI do have some local tweaks to Is it possible you tested against Krastorio and not Krastorio 2? I haven't tried with Krastorio since it's not compatible with 0.18. |
So, you need unpack Krastorio2_0.9.99.zip to mods directory or in any another directory ( in this case set option |
|
Ohh, OK. Sorry I'm blundering around here, I will try that the next chance I get. |
|
I'm making fairly steady progress with the Krastorio 2 mod, largely the issues I've been encountering is for producers in Krastorio, there are multiple unexpected combinations of group and subgroup (smelting producers that are actually assembling machines, assembling machines that are actually furnaces). I'm currently stuck on item names, as it seems loc_name isn't filled in for many items (it's using the entity name instead of the item name, and entity name looks more like a coded id). I think many of the recipes are also going to pose problems because there are many that, for example, output "0" of a "void product". But I haven't gotten to those yet. If you have any ideas on that I'd love to hear it, otherwise I'll keep playing with it later. |
|
I've added fix to handle strings like Recipes with zero result, as I understand it, apply to some kind of machine that heats liquid. but I could not determine what these machines are. Can it be simple to exclude such "zero" recipes? |
|
Аdded a json based demo page for 0.18.36 + Krastorio2, I think it will be a little more clearer. https://hhrhhr.bitbucket.io/factorio_lab_tools/0.18.36_krastorio2.html p.s. |
|
That's pretty cool! Do you know what the deal is with the smelting-crafting category? Everything under that tab also appears under intermediate products, and when I ran the app, that tab ended up showing as empty (I don't currently support showing things in two tabs). Would it make more sense for me to convert the "row" number field in the data to something like "subgroup", since that's a more direct correlation to what's in the data? I don't think it would be a major effort. I haven't tried out your fixes for the I was also thinking it might be easiest to simply exclude the zero-result recipes. The end result is they show something like: "out": {
"kr-void": 0
}It doesn't make any sense to me to include recipes like that. |
|
Recipes from group smelting-crafting shown in
And, judging by the recipe ID, these are different recipes from the point of view of the game. Using a "Iron beam" as an example (quoted from data.raw): Option I should probably try to fix the
|
|
Now recipes with
subgroup for items also exported to json. |
|
Help me out here, it's still missing names for |
|
Yes, there is some weirdness with this equipment. Localised strings from game: But I get it: As I understand it, this is due to the fact that in the mod localized strings are described in the data.raw ["energy-shield-equipment"] section, and not in data.raw.recipe or data.raw.item, where the script looks for such strings. I will think about how to determine the correct sequence for extracting localized strings. |
|
Thanks. It seems like it may have something to do with possibly changing the ids of those items as well, there are localized strings for |
|
@hhrhhr are you able to build for Bob's & Angel's? I'm running into problems left and right. I had to copy |
|
angelsaddons-cab_0.2.3 only — no problem: Without a specific error log, it is difficult to identify the cause. There are no errors when using Lua with patches from Factorio developers. But the usual Lua 5.3 gives me this error: I suppose this is because in Lua 5.3 by default all numbers are of type double, and the mod author is trying to format them as integers. |
|
Sorry, that's fair. The error you ran into I worked around by setting |
|
Try to use p.s. |
|
Hey @hhrhhr, I've actually been able to make a lot of progress with Bob's and Angel's, at this point it seems like the only issue is that some of the icons don't get resized the way they should, which is odd to me. You can see the current sprite sheet here. It looks like some of the Angel's sprites are messed up in row 43. I checked the output of your tool and it didn't list any problems in that row, although for a lot of the icons I get different messages like:
I know some of these don't matter but some seem new to me. I did use the -v flag with this run. local function apply_tint(ico, tint)
local X, Y = ico:sizeXY() -- gd func
for y = 0, X-1 do
for x = 0, Y-1 do
local c = ico:getPixel(x, y)
if tint.r ~= nil and tint.g ~= nil and tint.b ~= nil then
local r = math.floor(ico:red(c) * tint.r)
local g = math.floor(ico:green(c) * tint.g)
local b = math.floor(ico:blue(c) * tint.b)
-- local a = math.floor(ico:alpha(c) * tint.a)
local a = ico:alpha(c)
c = ico:colorAllocateAlpha(r, g, b, a)
if not c then
local orig = ico:getPixel(x, y)
if orig ~= nil then
c = orig
end
end
end
ico:setPixel(x, y, c)
end
end
endSadly it takes like 30 minutes to generate this sprite sheet also :( |
|
I have not found a way to disable such warnings other than modifying the libpng sources (this is wrong way). If there are many such warnings, then it is best to warn the author of the modification about using outdated tools. Or you can just redirect the stderr output to a file and hide it ) Regarding improperly scalable icons - what mod are they from? I can also see that the overlay icons at this scale are rendered too small too. Probably, there is an error somewhere due to 128px icons from the mod, while all calculations are based on the fact that icons can be at most 64px. So I'm waiting for the name of the mod ) |
|
It seems to be some of angel's, angel's refining is one with a problem and i can duplicate the issue with just that mod. I think you're on the right track suspecting the 128px but haven't had enough time to troubleshoot. |
Need to build a utility that scrapes the Lua game data and builds it into the JSON format this app expects. This would be the first step towards supporting mods.
The text was updated successfully, but these errors were encountered: