[BUG] "Scan game journals for exobiology" does not find files

[poinck]

The settings options "Scan game journals for exobiology" does not find files and therefor fails to update status.

Steps to reproduce

1. Go to "Files > Settings > AST"
2. Click on "Scan game journals for exobiology"
3. See error

Exception in Tkinter callback
Traceback (most recent call last):
  File "/usr/lib/python3.10/tkinter/__init__.py", line 1921, in __call__
    return self.func(*args)
  File "/home/poinck/.local/share/EDMarketConnector/plugins/ArtemisScannerTracker/load.py", line 439, in buildsoldbiodatajson
    self.rawvalue = build_biodata_json(logger, config.default_journal_dir)
  File "/home/poinck/.local/share/EDMarketConnector/plugins/ArtemisScannerTracker/journalcrawler.py", line 64, in build_biodata_json
    edlogs = [f for f in os.listdir(journaldir) if f.endswith(".log")]
FileNotFoundError: [Errno 2] No such file or directory: 'None'

Expected behavior:
Update the exobiology status on main EDMC window

System:

• OS: Linux
• AST Version: 0.2.5
• EDMC Version: 5.8.1

Additional context:

• I can rule out that EDMC might not find the journal files, because other plugin do find them.
• Could it be that some of the json files that track the bio-status are not there yet?
• I had to restart EMDC twice, because first time AST did not find it's own config, after clicking "OK" in the settings menu and restarting it, it started to work.

[Balvald]

I wondered when I get the first os specific problem.

First impression: It seems that the journaldir is None.

I can rule out that EDMC might not find the journal files, because other plugin do find them.

thats peculiar because with the mentioned button I use the folder specified in EDMC config.
I give config.default_journal_dir to the build_biodata_json function and it most probably rules out that the E:D journal file location isn't correctly set in the Configuration tab.

edlogs = [f for f in os.listdir(journaldir) if f.endswith(".log")]

Anything that could go wrong:
journaldir = config.default_journal_dir is "None". if it is.
os.listdir(journaldir) is "None" but journaldir isn't. Maybe weird formatting.
or the issue is with f.endswith(".log"). Maybe weird formatting.

I'll hope its not one of the latter ones.

@poinck does it work with the other button, when you got journal files at /home/.../EDMarketConnector/plugins/ArtemisScannerTracker/journals/?

if that works I can essentially rule out the latter two.
if it is that way then as a apply yourself hotfix until I come around to do it:

Code to modify (starts at line 434 at load.py):

def buildsoldbiodatajson(self) -> None:
    """Build the soldbiodata.json using the neighboring journalcrawler.py."""
    # Always uses the game journal directory

    global logger
    self.rawvalue = build_biodata_json(logger, config.default_journal_dir)

to:

def buildsoldbiodatajson(self) -> None:
    """Build the soldbiodata.json using the neighboring journalcrawler.py."""
    # Always uses the game journal directory

    global logger
    journaldir = "<insert path here, when using \ might need double \\ as \ is escape char>"
    self.rawvalue = build_biodata_json(logger, journaldir)

I do not know what the path should be on a linux system, I just hoped config would know.

Bear with me with the fix. I'm currently in an exam phase in my studies and might take a bit longer.

[poinck]

I copied over all my Journals to ~/.local/share/EDMarketConnector/plugins/ArtemisScannerTracker/journals and did the thing. It did not complain of any missing or not found directories or files, but it only found the 4 bios from today's session. I am out of reach of any station to sell at Vista and I did not sell any for weeks. So, there should be more in it.

[poinck]

I checked with my own plugin and how I access the game journals and found this:

def journal_entry(
    cmdr: str, is_beta: bool, system: str, station: str, entry: Dict[str, Any], state: Dict[str, Any]
) -> None:
    if entry['event'] == 'StartJump':
        # ...

It seems, I never access the files directly. But this is not an option in this case, because, all past events need to be read. I think, it has to be something about escaping spaces in the path. I'll try the hotfix and report back.

[poinck]

Interesting, with this:

diff --git a/load.py b/load.py
index 5fe1c55..988bf48 100644
--- a/load.py
+++ b/load.py
@@ -436,7 +436,10 @@ class ArtemisScannerTracker:
         # Always uses the game journal directory
 
         global logger
-        self.rawvalue = build_biodata_json(logger, config.default_journal_dir)
+        print(config.default_journal_dir)
+        jdir = "/home/poinck/.var/app/com.valvesoftware.Steam/.steam/steam/steamapps/compatdata/359320/pfx/drive_c/users/steamuser/Saved Games/Frontier Developments/Elite Dangerous"
+        self.rawvalue = build_biodata_json(logger, jdir)
+        #self.rawvalue = build_biodata_json(logger, config.default_journal_dir)
 
 
 # region eventhandling

.., I get the same result like with the "local" copy of the journals.

None
{'species': 'Fonticulua Digitos', 'system': 'Blaa Dryou US-Z b1-0', 'body': 'Blaa Dryou US-Z b1-0 6 g'}
{'species': 'Stratum Paleas', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}
{'species': 'Aleoida Laminiae', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}
{'species': 'Bacterium Alcyoneum', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}

I have very old Journal files in this directory. Never lost a single one since I play ED. Could old journal formats be the trouble makers?

[Balvald]

Alright, config.default_journal_path seems to be None then.
That the local copy does the same for the same set of journals is to be expected.

old journal formats shouldn't matter with the amount that I tested the journalcrawling against.Though the possibility that it is, is not 0. If it generates these 4 when before the jsons were newly generated by the plugin it won't be because of old journal formats. When its a problem with journals not being included fully it won't even update the jsons.

We might have another bigger issue here.

The journal crawling seems to run and finish as normal

Assuming os.listdir(path) gives every filename in the path and f.endswith(".log") works - essentially if edlogs mentioned in a code block of my first comment on here in journalcrawler.py does have a list of all journal file names then theres something wrong with the journalcrawling itself instead.

Where I'd be very thankful to get my hands on the journals that make the journalcrawling misbehave.

[poinck]

Would it be enough to give you the journals of the last 3 weeks or so? These should be journals you can be sure of that they include scanned bios.

And another thought: Could file sorting be an issue?

[Balvald]

Would it be enough to give you the journals of the last 3 weeks or so? These should be journals you can be sure of that they include scanned bios.

Yeah thats enough. As long as the bios in them don't track properly, then they are of interest.

And another thought: Could file sorting be an issue?

If it does not go through them chronologically then yes. It's indirectly hinted at in the readme that a missing journal file will create weird trackings when using the journalcrawler.

Its something I check everytime and so far it has always done them chronologically.

Anyway If I get different behaviour on my machine. (i.e. "Works on my machine") I'll be configuring a whole Elite Installation on a fresh linux install to test against.

Regarding that config.default_journal_dir is None I've found that it seems intentional? as config for linux inits:
self.default_journal_dir_path = None # type: ignore
and then does nothing with it later.
Since the whole things still can use and access the journals albeit only the newest. it obviously must know where it can find them. might be that what I used is a fallback that does not make sense in a linux context.

[poinck]

It took me a while, but here there are:

Journal.2023-01-01T153328.01.log
Journal.2023-01-02T183615.01.log
Journal.2023-01-05T185954.01.log
Journal.2023-01-07T164113.01.log
Journal.2023-01-08T150735.01.log
Journal.2023-01-12T180531.01.log
Journal.2023-01-13T185212.01.log
Journal.2023-01-14T151121.01.log
Journal.2023-01-15T150537.01.log
Journal.2023-01-19T182704.01.log
Journal.2023-01-22T172749.01.log
Journal.2023-01-29T162329.01.log
Journal.2023-01-30T170735.01.log
Journal.2023-01-31T115753.01.log
Journal.2023-02-05T150332.01.log
Journal.2023-02-05T150955.01.log
Journal.2023-02-05T151910.01.log
Journal.2023-02-05T152608.01.log
Journal.2023-02-05T153725.01.log
Journal.2023-02-05T155025.01.log
Journal.2023-02-05T160138.01.log
Journal.2023-02-05T160524.01.log
Journal.2023-02-05T161144.01.log
Journal.2023-02-05T161502.01.log
Journal.2023-02-05T165627.01.log
Journal.2023-02-05T180341.01.log
Journal.2023-02-07T095956.01.log
Journal.2023-02-07T152840.01.log
Journal.2023-02-11T155658.01.log

[Balvald]

on windows I just now couldn't reproduce the behaviour.

the respective soldbiodata.json and notsoldbiodata.json I get out of them:
notsoldbiodata.json.log
soldbiodata.json.log

it seems to be able to work through the set you send me just fine.

Though peculiar is I do not see any mention of these four:

{'species': 'Fonticulua Digitos', 'system': 'Blaa Dryou US-Z b1-0', 'body': 'Blaa Dryou US-Z b1-0 6 g'}
{'species': 'Stratum Paleas', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}
{'species': 'Aleoida Laminiae', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}
{'species': 'Bacterium Alcyoneum', 'system': 'Thailooe RB-X d1-1', 'body': 'Thailooe RB-X d1-1 C 5 a'}

and I checked the journals there is no mention of them, they have been done earlier?

Assuming that its not os.listdir() that could sometimes not give out not already presorted under some circumstances.
There be something weird In between when these four were scanned/sold and the point in time the set of journals you've send.

(os.listdir() being the culprit is much more probable than I thought before since the order of the output of os.listdir() may depend on the filesystem, got no proof of that myself yet, but seems reasonable to assume that can be the case anyway.)

@poinck:
Though before I'll send you to search please test the current state of this branch: https://github.com/Balvald/ArtemisScannerTracker/tree/45-bug-linux
to check if the assumption its os.listdir is actually the problem, then a single sort should fix the problem or if the problem is seated in the journals form that interval between the only 4 signals that got tracked and the earliest journal you provided.

[poinck]

I can test this branch (did the checkout). The thing is, I recently sold all my bio data. I need to think of a way to test this now. I think I won't be doing exobiology for some weeks. I suppose, I can just add the files I'be sent you to the journal folder of the plugin and click the button.

[Balvald]

just try it with the gamejournal button on that branch. if all is sold they all should appear in soldbiodata.json instead of notsoldbiodata.json which should end up mostly empty.

[poinck]

Oh, it might have worked!

I assume, that I need to click that RESET button before the next bio scan.

[poinck]

Ok, it seems I confused it now with the plugin journal folder, because it still reports (a different sum) of unsold bio data after clicking the button for the game journal (but still less than I actually got, even without first-discovery bonus).

[Balvald]

whats the state of the notsoldbiodata.json then? I remember calculating the unsold value off of it somewhere in the code.

[Balvald]

even when pressing these buttons several times it should sort itself and not end up with anything in notsoldbiodata if everything was sold and or lost.

[Balvald]

would like some closure on thise one if the notsoldbiodata.json ended up appropiately empty on your side sometime. @poinck So I can close this in good conscience when the branch gets to main.

[Balvald]

Couldn't find anything wrong with it right now.

Ok, it seems I confused it now with the plugin journal folder, because it still reports (a different sum) of unsold bio data after clicking the button for the game journal (but still less than I actually got, even without first-discovery bonus).

Still sounds to me that there is something fishy going on but the original problems to these seem fixed.

As of now I can't get it to misbehave and report a higher value than it should be in the unsold value without leaving journal entries out.
I'll created an issue #48 where anyone'd be welcome to give hints to such cases.

I'll close this one as changes are on dev now.

[poinck]

Thx for everything so far. I will monitor #48 and contribute to it, if I see something reproducable.