[BUG] Investigation into misbehaving journalcrawling

[Balvald]

Describe the bug
Unsold scanned value misreporting after using the journal scan buttons in the settings tab.

To Reproduce
Steps to reproduce the behavior:

1. Go to AST tab in settings
2. Click on "Scan game journals for exobiology"
3. plugin reports unsold exobiology
4. go to vista genommics and try to sell
5. See that there is actually no unsold exobiology left!

OR

1. Go to AST tab in settings
2. Click on "Scan game journals for exobiology"
3. plugin reports unsold exobiology
4. go to vista genommics and try to sell
5. See that there is unsold exobiology but the amount reported is wrong!

Expected behavior

1. Go to AST tab in settings
2. Click on "Scan game journals for exobiology"
3. plugin reports unsold exobiology
4. go to vista genommics and try to sell
5. See that there is unsold exobiology left to sell.

OR

1. Go to AST tab in settings
2. Click on "Scan game journals for exobiology"
3. plugin reports no unsold exobiology
4. go to vista genommics and try to sell
5. See that there is actually no unsold exobiology left.

Desktop (please complete the following information):

• OS: Any
• AST Version: 0.2.5, 0.2.6 and onwards
• EDMC Version: 5.8.1 (shouldn't matter on this one honestly.)

Additional context
Please add cases where this happened, if this happened to you.
Preferably with a set of continous journals that trigger said behaviour.
(Meaning there is no information missing on what the character has done in between. Important are only biodata scan events, deaths, biodata sell events.)

[Balvald]

Oh, it might have worked!

I assume, that I need to click that RESET button before the next bio scan.

That its not showing 0 Cr. (in this case because everything was sold) is unintentional behaviour.

[Balvald]

taken from #45

[Balvald]

Gotta start documenting on new lead on this one:
Started with this forum posts.
https://forums.frontier.co.uk/threads/artemis-scanner-tracker-a-edmc-plugin-that-keeps-track-of-the-exobiology-scanner-of-a-cmdrs-artemis-suit.612191/page-2#post-10199323

Data was provided. I was able to reproduce the behavior c:

Weird thing both the edmc log that was provided as well as my own edmc log while reproducing recognizes that plants have been scanned near the end of the journal crawling.
(so any of the 3 possible states log, sample and or analyzed)
I specifically confirmed that "Analyse" Scan Organic events were infact in the journals.

Most possibly the problem: In the edmc logs it states that certain horizons (3.8) logs from 2021 got read after the newest ones. one of which includes the death of the CMDR!
Meaning any of the plants that were logged will just vanish.

This would also explain why I couldn't reproduce it with the old dataset because that one did only include the most recent journals that got read perfectly chronologically and they were all odyssey ones from what I remember. (I feel a bit dumb now at this oversight that which is techncally creating problems in this project since the journalcrawling.py existed.)

Things that need change to fix this:

• The plugin should have a debug string for when we read an Analyse type Scan Organic Event they are infact the only ones that really matter within the context of the journal crawler. (will make the dev/debug experience better in the long run)
• Unclear why the journal crawling still goes through all log files that are created from the horizons (3.X) version of the game. (should limit it since I know now it invites unexpected behaviour)
• I should limit it to 4.X only with the exception of 3.X versions of horizons for a specific date range from release of Odyssey until the different game versions were splitted (I'll have to check on which date and time those have happened)
• The info from that short span from 3.X is only needed to check when the player may have died and possibly lost exobiology data and location changes for a minimum of resilience (this will still prompt "Key error messages" in in the debug log because off how the different location based events in the journal were structured prior to odyssey.)
• Any other 3.X logs add no value to the tracking.

And I have to check again that from this subset of journals with different name conventions that the journal crawler goes through them chronologically. (like that was the whole point)

[Balvald]

List with tick boxes for the things that need changing:

• The plugin should have a debug string for when we read an Analyse type Scan Organic Event they are infact the only ones that really matter within the context of the journal crawler. (will make the dev/debug experience better in the long run)
• I should limit it to 4.X only with the exception of 3.X versions of horizons for a specific date range from release of Odyssey until the different game versions were splitted (I'll have to check on which date and time those have happened)
• I have to check again that from this subset of journals with different name conventions that the journal crawler goes through them chronologically.

[Balvald]

The version on the investigation branch now has the underlying problem fixed.

Still have to add the check to ignore journal files of the 3.X version if they are not from 2021-05-18 till 2022-11-30.
the dates themselves are one day before Odyssey launched and one day after 4.X and 3.X versions completely disconnected from each other.

[Balvald]

Things done. gonna move it over to main now.