Using -stay_open feature of ExifTool

[Clariden]

Hi Viktor,

I know, timing's bad ;). You're busy with the next release and the holiday season's just around the corner. But I had a thought I wanted to share before I forget it. I saw in your code that you're starting ExifTool anew for every task. I think, it may be better to start Exiftool with the -stay_open switch when starting GTN and leave it open until GTN is closed. As a Perl script, Exiftool takes a very long time to be loaded and compiled. If you leave it open, you can avoid this overhead. You communicate with Exiftool via stdin or via an argfile and get an--almost--lightning fast response. Maybe this could be a starting point for the implementation.

Happy holidays!

[nemethviktor]

Hey, no worries -- so yes depending on what version of the app you're looking at it's generally correct.
TLDR was that up until the release that I think went out today (to Master, not to Dev, that's been the "new" case for a while) I had been using a nuget module asyncExifTool (which used stay_open) to do the things but i found that if I actually parse whole folders via CSV in/out it's a lot faster than using the add-in. So I haven't really looked at it since. I'll have a second look in the new year though.
Thanks for the link!

[Urmel]

Hi Victor,
I fiddled a bit around with it using the ExifToolWrapper. It seems to be quite quick - at least when parsing dirs with a smaller number of files (maybe this changes when directories get very large....)
I will try to implement this completely and provide a PR (sep. branch?) so that we can test....

[nemethviktor]

Feel free to experiment ;)
That said I think the current CSV logic is likely to be as fast as it gets - if we could have the stay_open somehow enabled for the process though while keeping the CSV logic it would possibly be a winner.

[nemethviktor]

Also also remember that non standard English characters (umlauts and Asian ones) cause a lot of problems via CMD and many other things whereas CSV is already UTF8

[Urmel]

Hi @nemethviktor,

I have a running implementation which is quite fast (for my test cases) and it handles sidecar files, respects umlauts / UTF8 (at least for the bit I tested) and shows the same result on my test dir which your latest build shows (comparison see PR). I've also briefly tested using the editform - works as well.

The PR #52 includes

• Using the EXIFWrapper (stay_open) to gather EXIF data when reading in a directory
• Configuration (what tags in which order to what property) now in code
• Putting all that data in the model / directory elements and refreshing the list view from that

There are some open topics, as documented in the PR.

How would you like to proceed... ?

[nemethviktor]

Hey, thanks for the hard work on this! :) -- I'm off for the weekend but will have a proper look next week.
I haven't tested it so far but I have a "test bed" sorta VM that has a folder structure such as

• c:\temp
• c:\temp_őő
• c:\temp őő北京

and then in each folder files that are named with similar logic (umlauts and/or Chinese chars). At the moment (the current public release) reads files from each of those folders. (Admittedly there is an issue w saving the xmp file in the last folder espc if it also has Asian chars but since nobody complained about it so far I haven't really looked.) -- anyway, if you feel like playing around with it, see if stuff works (reading) w the logic above? - otherwise I'll look early next week.

thanks again ;)

[nemethviktor]

Hey @Urmel , gotten around to have a look at this and thank you again!

A few comments in no particular order

• [wish] image previews should also be called by the stay_open version. Basically most pics are likely to require a generated preview so firing up ET each time when we have this already working seems like a no-brainer for switch
• [wish] the "processing file" label seems to be using lower cases. Seems a bit odd but not the end of the world.
• [question] i see you have built a model for the exif tags and priorities. can we then remove (at least part of) the sqlite file that had been powering this in the past? (though i recall the write [as in read/write] methods/functions might still use some of it)
• [question] - would it be possible to async fire up ET as soon as the app is loading? appreciating this hasn't ever been the case but now that we have the current (new) logic i think if ET could be started early then the loading of the initial folder would appear faster to the user?
• [comment] listing a large number of folders seems to be a bit slower as they now appear one by one rather than batch. i think this has less to do with ET as it'd not be called for this task but what I used to have was an application.doevents() on every 5th or 10th folder i think - i admittedly skimmed through that part of the code but i think part of the logic could be made faster if the folders were put on the screen in batches.
• [bug] tried running the version on English/Hungarian/German VMs but it's not related to language per se. The 'dot' is missing from filenames. e.g. instead of seeing 20071012_Lake_District_EOS__MG_7718.CR2 i have 20071012_Lake_District_EOS__MG_7718cr2 - I think this has to do with how Windows Explorer shows the files, which is also being pulled into GTN - that has always been the case (it being pulled into the app) but I think the recent changes undid some logic I put in to counter this issue in the past. Basically if WE does show the extension then stuff works fine, but if it doesn't, it won't. See below for screenshots.

Good:

Not Good:

... this then causes a crash when trying to Edit the file because the file with the text value of the listViewItem doesn't actually exist.

Otherwise awesome :) -- thank you :)

[nemethviktor]

Hi @Urmel ,

+1 bug -> "negative" lat/long values are not being respected/loaded. E.g. the file copied here should have -2.451665.... for longitude and as such should point to some place in the UK but in the current version it's not-negative 2.451665 and thus points to the middle of the North Sea.

Also on the note of "Edit Form crashing due to incorrect file naming" -- that also affect the loading of previews, which doesn't work if the Windows Explorer setting is as described in the previous post.

[Urmel]

Hi @nemethviktor,

thank you for the extensive test report!!
Here some fixes and responses...

ImagePreviews with StayOpen

[wish] image previews should also be called by the stay_open version. Basically most pics are likely to require a generated preview so firing up ET each time when we have this already working seems like a no-brainer for switch

Fully with you there!

• I will create a new feature request for that.

"processing file" with LowerCase

[wish] the "processing file" label seems to be using lower cases. Seems a bit odd but not the end of the world.

Yep - I used lower case for comparison with the XMP filename. But you are right - that should not be visible to the user! Fixed.

Remove "in" part of EXIF Mapping from SQLite

[question] i see you have built a model for the exif tags and priorities. can we then remove (at least part of) the sqlite file that had been powering this in the past? (though i recall the write [as in read/write] methods/functions might still use some of it)

I'm afraid so not - the Track Sync uses the same Exif logic as the Folder Parsing used previously. We will have to remove that first.

• I will create a new feature request for that.

Async Start of ExifTool

[question] - would it be possible to async fire up ET as soon as the app is loading? appreciating this hasn't ever been the case but now that we have the current (new) logic i think if ET could be started early then the loading of the initial folder would appear faster to the user?

I now made it start together with the app - so it still loads, but only once. For every subsequent folder click, it is reused. From an effect effort perspective, this seemed to be a good compromise...

Large Number of Folders Slower

[comment] listing a large number of folders seems to be a bit slower as they now appear one by one rather than batch. i think this has less to do with ET as it'd not be called for this task but what I used to have was an application.doevents() on every 5th or 10th folder i think - i admittedly skimmed through that part of the code but i think part of the logic could be made faster if the folders were put on the screen in batches.

I seem not to have a comparable test setup - what's "large" :)
Anyway, I reduced the update frequency for folder parsing (label text) and adding items to the list view (doevents).

Missing dot before extension (also crashes FrmEdit and Preview)

[bug] tried running the version on English/Hungarian/German VMs but it's not related to language per se. The 'dot' is missing from filenames. e.g. instead of seeing 20071012_Lake_District_EOS__MG_7718.CR2 i have 20071012_Lake_District_EOS__MG_7718cr2 - I think this has to do with how Windows Explorer shows the files, which is also being pulled into GTN - that has always been the case (it being pulled into the app) but I think the recent changes undid some logic I put in to counter this issue in the past. Basically if WE does show the extension then stuff works fine, but if it doesn't, it won't. See below for screenshots.

Nice catch! I actually removed the "dot" from the extension property of the directory element. So it got left away when adding the extension in case there is no extension shown in WE... Fixed. This also seems to have fixed the crashing editform in the case and the failing previews...

Longitude Ref not respected

+1 bug -> "negative" lat/long values are not being respected/loaded. E.g. the file copied here should have -2.451665.... for longitude and as such should point to some place in the UK but in the current version it's not-negative 2.451665 and thus points to the middle of the North Sea.

Shame on me - copy paste issue :( Was using the LatRef entry as LongRef value... Fixed.

[Urmel]

New Feature Requests:

• Use EXIFTool Available to also Generate ImagePreviews #55
• Migrate GPXTrackSync to use EXIFWrapper and Remove ObjectExifIn-Mapping #56 -> for this, we should understand if another step is needed beforehand.

[nemethviktor]

Particular to Large Number of Folders Slower --> parsing the list of 266 subfolders (no files at all) takes about 6.9 seconds. In the current dev version it's 1.1 sec. :)