Access environment variables with format strings

[JessiAuro]

When downloading files and/or running exec scripts, it would be nice to have some kind of guaranteed unique value tied to a specific call of gallery-dl.

Ideally, each run would generate a GUID and make it accessible with something like {_instance} (like other values in the exec post-processor).

Additionally, a field containing a datetime of when gallery-dl was called would be nice too.

[Hrxn]

Not sure, I mean adding UUID generation would probably not be much of a thing, but isn't this something that's best handled by your shell? Simply use a short script to call gallery-dl, and you can easily access any UUID from within.

In bash, you can either use uuidgen or something like UUID=$(cat /proc/sys/kernel/random/uuid), I think.
PowerShell works as well, by using the built-in New-Guid.

[JessiAuro]

I do have a wrapper script for this, it would just be nice to have access to such a value internally, to use for building file paths and passing to post-processor scripts. Essentially as a job ID I can use in my paths to separate out files generated by individual runs of gallery-dl.

Perhaps this would be better implemented as allowing you to reference environment variables in format strings, so I can set/use any variable I want easily from the shell. Something like {env%HOME!<conversion>:<format>

[mikf]

You can add custom entries to each metadata dict with the keywords option.
gallery-dl -o keywords._instance=$(uuidgen)

But you are right, having access to environment variables and other data in format strings would be nice.

[JessiAuro]

You can add custom entries to each metadata dict with the keywords option.

Ah my bad, I totally missed that in the docs. The docs say extractor.*.keywords but the command you give simply uses keywords.

Does the -o option presume the extractor namespace or can any option be specified in the "root" of the options namespace?

[mikf]

Any extractor option can be used as

1. <option>
2. extractor.<option>
3. extractor.<category>.<option>
4. extractor.<category>.<subcategory>.<option>

Just <option> would be the global namespace that overrides all settings anywhere else, i.e. option priority would be 1, 4, 3, 2. Global namespace is also what gets used by command-line options.

This is all somewhat described here.

or can any option be specified in the "root" of the options namespace?

Most can. Output options and something highly specific like extractor.ytdl.enabled must be in output or that particular place.

[JessiAuro]

The docs doesn't seem to mention the existence of a global namespace/scope though, might be a good idea to update it to something like this:

Scope	Priority	Syntax
Global	1	<option-name>
Extractor	4	extractor.<option-name>
Category	3	extractor.<category>.<option-name>
Subcategory	2	extractor.<category>.<subcategory>.<option-name>

And then perhaps options could specify the "scope" they are valid at.

[Hrxn]

[..]

But you are right, having access to environment variables and other data in format strings would be nice.

So, basically something like gallery-dl's Path type, which already should support environment variable expansion, except not only for paths?

[Hrxn]

The docs doesn't seem to mention the existence of a global namespace/scope though, might be a good idea to update it to something like this:
[..]

Yeah, well, maybe not directly, but it clearly explains the specificity of the settings for the configuration file. Calling it a global namespace, while correct, might be a bit misleading, so this is something that could be considered. But here's this part of the main README that mentions setting options on the command-line overrides any settings defined per configuration, just like what is considered usual best practice for all CLI apps.

[mikf]

So, basically something like gallery-dl's Path type, which already should support environment variable expansion, except not only for paths?

I thought of doing it more like url-metadata and parent-metadata where the user can specify a name and any metadata dict will have a dict-like object with environment variables with that name.

"environment-metadata": "_env" -> "{_env[HOME]}"

Also, if any of you have a better suggestion for the option name, I'm all ears.
I think eventually all of these should be renamed to metadata-... where ... describes the field's function.

[ImportTaste]

Oh my god, thank you. This is going to be super useful.

[mikf]

So yeah, environment variable access is now possible in all format strings and without enabling any options with {_env[NAME]}.

Adding a UUID/GUID ({_uuid}?) or a datetime object representing the current time ({_now}, {_time}?) would be easy enough as well, so if you want I can add that too.

[JessiAuro]

Sweet! This will simplify a lot of things, thank you.