Display conversion progress

[heinezen]

Required skills: Python

Difficulty: Easy

For the initial asset conversion, openage provides textual output to signal which stage of conversion has been reached. The output usually looks like this:

INFO [py] converting metadata
INFO [py] [0] palette
INFO [py] [1] empires.dat
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] Dumping metadata files...
INFO [py] [2] player color palette
INFO [py] [3] terminal color palette
INFO [py] [4] game specification files
INFO [py] asset conversion complete; asset version: 7

However, some of these stages can take a while to complete. For example, the whole Exporting media files... stage can take up to 20 minutes. For these stages, it would be nice to have more dynamic updates in form of a progress bar. This would also give much more feedback to users on what's being done at the moment.

The stages that would benefit the most from a progress bar are:

• empires.dat file parsing (See code)
• Creating nyan objects... (See code)
• Exporting media files... (See code)

We provide a formatter for the progress bar in our tools: https://github.com/SFTtech/openage/blob/master/openage/util/strings.py#L103

Further reading:

• Converter architecture
• Converter workflow

[jochenrui]

hi there, can I try this for my first issue ? :)

[heinezen]

@jochenrui Sure, contribution is welcome :)

If you have any questions not answered by the issue, you can also come by in our chatroom (see the project readme for infos).

[jochenrui]

@heinezen
I actually have no idea where to start. Is there a way to run the initial asset conversion stage in isolation?

Also the progress bar is supposed to reflect the actual progress right? So I'd have to divide the total work to be done by e.g. 100 right?

I assume in the read signature the size of "raw" can be used to determine the progress?

def read(self, raw, offset, game_version, cls=None, members=None):

same for fulll_data_set in create_nyan_objects?

def _create_nyan_objects(cls, full_data_set)

and count of export_requests in export?

def export(export_requests, sourcedir, exportdir, args):

[heinezen]

@jochenrui Ah yes, you can run the converter in isolation. After you built the project, the command

bin/run convert

will run just the converter. I would recommend

bin/run convert --force --devmode --source-dir <path-to-AoE-install>

because that skips the initial prompts and jumps straight to the conversion. The --no-media flag is also helpful because media conversion can take a long time. bin/run convert --help shows other options that may or may not be helpful.

I assume in the read signature the size of "raw" can be used to determine the progress?
same for fulll_data_set in create_nyan_objects?
and count of export_requests in export?

You got the right idea :)

raw is just the file in a bytearray, so size(raw) should give you the file size.

[jochenrui]

@heinezen thanks for your very in-depth comment. It's very reassuring, and sorry for my slow progress. Work has been stressful these days. I'll get to it right now :)

[jochenrui]

@heinezen have you seen this error before during configuration? I followed the instructions on https://github.com/SFTtech/openage/blob/master/doc/build_instructions/macos.md

CMake Warning at buildsystem/HandleCXXOptions.cmake:75 (message):
  GNU gold linker isn't available, using the default system linker.
Call Stack (most recent call first):
  buildsystem/HandleCXXOptions.cmake:107 (try_enable_gold_linker)
  CMakeLists.txt:172 (include)




The compiler doesn't support required C++20 features:
  * Concepts
  * Default comparisons
The following versions support these features:
  * clang++ >= 10
  * g++ >= 10
  * Microsoft Visual Studio 2019 >= 16.8
Please upgrade your compiler to build openage.


CMake Error at buildsystem/CheckCompilerFeatures.cmake:57 (message):
  aborting
Call Stack (most recent call first):
  CMakeLists.txt:173 (include)

[heinezen]

@jochenrui I think the macOS clang compiler doesn't support C++20 features yet. Can you try with gcc?

Edit: Or alternatively with pure clang from homebrew.

[jochenrui]

@heinezen I'm already using pure clang from homebrew I think

 ./configure --compiler=clang

[jochenrui]

@heinezen gcc doesn't work either :/

./configure --compiler=gcc

[heinezen]

@jochenrui Sigh It's always Apple (we can't test macOS well because it does't run in VMs and hardware is expensive :D). I'll take a look at the problem again.

You could try Ubuntu in a VM as a temporary workaround...

[jochenrui]

@heinezen sorry about that. I'll try to get it running on macos first, maybe this would make for a good addition to the build_instructions file for macos :D

[heinezen]

@jochenrui I may have found solution if this bug is responsible.

1. Clone our database code nyan if you haven't.

2. Rename the version file in the repository to nyan_version

3. Back in the openage repository: Run make cleanbuilddirs

4. Comment out

   openage/CMakeLists.txt

   Line 173 in 85ac46e

   include(CheckCompilerFeatures)

   to see macOS compiler errors.

Don't know if that works or if that's just a different problem.

[jochenrui]

@heinezen I'm already installing a ubuntu vm, I gave up on getting it going on macos like a minute before your message ^^

[heinezen]

@jochenrui Well, that's fine by me then :)

[TobiasFP]

If you get stuck or need inspiration, there has been made a dockerfile that uses an Ubuntu image to build the system, which could be of use:
#1357

[heinezen]

@jochenrui And tell me if you need any game edition files for testing :)

[jochenrui]

@heinezen
is there a way to mock the game files? Btw I'm still building since about 2 hours lol

[jochenrui]

not building.. installing dependencies.. building is the next step lol

[heinezen]

is there a way to mock the game files?

Not right now because a lot of the AoE file formats are weird proprietary formats that are not that great to handle/create. Making mock files is a good idea though, but not the most fun task :D. The converter being able to convert any of the myriad of editions that exist by now for AoE1/AoE2 and SWGB must suffice for now..

[jochenrui]

@heinezen
in that case, whats the smallest game file u have :D

[heinezen]

AoE1 - Rise of Rome is the smallest with ~220 MB. (meet me in our chat for the files)

[duanqn]

@jochenrui If you want to try Windows, I have a successful build here: https://github.com/SFTtech/openage/runs/3883281983.
You can look at the YAML file for the exact steps.

Edit: Or just download the build artifacts :)

[maassen1]

Does this issue still need to be resolved?

[heinezen]

@maassen1 Yes, it's still not implemented.