Proposal: Have a tool that converts all the XML files to binaries for the client

[raduprv]

I don't think it makes much sense loading XML at runtime, especially since it very rarely changes.
I think it would be better if there was a tool that converted all the XML files to binaries, that are loaded at runtime by the client. This way we get rid of two dependencies (libxml2 and iconv), and would speed things up a bit, especially for slower devices, such as phones, raspberry pies, etc.

[gvissers]

LOL. I had some fun making a start on this. Check my dexml branch (https://github.com/gvissers/Eternal-Lands/tree/dexml) if you want to play.

Anyway, I thought I'd try to start by converting the XML to C code, which is then compiled into the executable directly. It saves the hassle of having to write import functions, and since we're dealing with text files it is much easier to debug. Optimistically I started with the actor definitions. Well, nearly two weeks and 200,000 lines of generated code later, I'm happy to say that the process worked. Of course the default (debug) build ballooned from 3.2M to 21M, and the overall speedup of the boot process is questionable, but it works.

So here's some numbers, five runs, all wall clock time. The time spent in init_actor_defs is indeed nearly halved.The variances in the overall boot time are much greater than any potential savings though. (well, I did only test it on my latop, which is reasonably fast and has an SSD. YMMV).

Actor defs parsed at runtime, default build:
Size of executable: 3.2M
init_actor_defs: 0.358 0.356 0.358 0.355 0.351
init_stuff: 1.687 1.211 1.613 1.656 1.255
avg init_actor_defs: 0.36s

Actor defs compiled in, default build
Size of executable: 21M
init_actor_defs: 0.165 0.166 0.181 0.207 0.177
init_stuff: 1.422 1.679 0.953 1.023 1.435
avg init_actor_defs: 0.18s

Actor defs parsed at runtime, release build:
Size of executable: 2.9M
init_actor_defs: 0.324 0.374 0.336 0.320 0.322
init_stuff: 1.414 1.026 1.005 1.161 1.115
avg init_actor_defs: 0.34s

Actor defs compiled in, release build
Size of executable: 7.9M
init_actor_defs: 0.151 0.140 0.160 0.153 0.144
init_stuff: 1.172 0.720 0.825 0.758 0.742
avg init_actor_defs: 0.15s

Anyway, now that I've hashed out how to store the data, I can more easily come up with a binary file format. For the encyclopedia we probably want to do this anyway, since I've just realized that that's 14M of XML data (as opposed to just 600K for the actor defs, though the latter includes a lot of its files multiple times for different actor types).

[raduprv]

Well, the good thing is that so far, the Android client takes far less time to load than I would have expected. Even on cheap tablets. So maybe it's not that necessary, although it would save a few seconds here and there.

…

On 2/24/2017 11:46 AM, Gé Vissers wrote: LOL. I had some fun making a start on this. Check my dexml branch (https://github.com/gvissers/Eternal-Lands/tree/dexml) if you want to play. Anyway, I thought I'd try to start by converting the XML to C code, which is then compiled into the executable directly. It saves the hassle of having to write import functions, and since we're dealing with text files it is much easier to debug. Optimistically I started with the actor definitions. Well, nearly two weeks and 200,000 lines of generated code later, I'm happy to say that the process worked. Of course the default (debug) build ballooned from 3.2M to 21M, and the overall speedup of the boot process is questionable, but it works. So here's some numbers, five runs, all wall clock time. The time spent in init_actor_defs is indeed nearly halved.The variances in the overall boot time are much greater than any potential savings though. (well, I did only test it on my latop, which is reasonably fast and has an SSD. YMMV). Actor defs parsed at runtime, default build: Size of executable: 3.2M init_actor_defs: 0.358 0.356 0.358 0.355 0.351 init_stuff: 1.687 1.211 1.613 1.656 1.255 avg init_actor_defs: 0.36s Actor defs compiled in, default build Size of executable: 21M init_actor_defs: 0.165 0.166 0.181 0.207 0.177 init_stuff: 1.422 1.679 0.953 1.023 1.435 avg init_actor_defs: 0.18s Actor defs parsed at runtime, release build: Size of executable: 2.9M init_actor_defs: 0.324 0.374 0.336 0.320 0.322 init_stuff: 1.414 1.026 1.005 1.161 1.115 avg init_actor_defs: 0.34s Actor defs compiled in, release build Size of executable: 7.9M init_actor_defs: 0.151 0.140 0.160 0.153 0.144 init_stuff: 1.172 0.720 0.825 0.758 0.742 avg init_actor_defs: 0.15s Anyway, now that I've hashed out how to store the data, I can more easily come up with a binary file format. For the encyclopedia we probably want to do this anyway, since I've just realized that that's 14M of XML data (as opposed to just 600K for the actor defs, though the latter includes a lot of its files multiple times for different actor types). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#34 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABC3X45YTEPnlMksAWan_ooCvKH4mlRJks5rfqcBgaJpZM4KQW3g>.