Config File Outline

The structure of the gallery-dl configuration file

Configuration files for gallery-dl use a JSON-based file format.

First steps

• If you're not familiar with it: Configuration file introduction, from gallery-dl's official README.
• Handling a JSON file: The JSON format is actually very simple.
  • Every somewhat decent text editor, no matter your operating system, should be able to handle it. You don't need explicit support for JSON, but having JSON syntax highlighting etc. definitely helps a lot.
• "My local text editor sucks!": There is an online (in-browser) version of the cross-platform editor VS Code.
  • https://vscode.dev/
  • Simply open a new file there (Main Menu (upper left corner), select File > New File; or use the New Text File button in Explorer View, right next to OPEN EDITORS), if you paste some JSON text into the editor it should automatically detect it and set the Language Mode accordingly.
    • If not: Set the Language Mode to JSON by hand: Either click on the field in the lower right corner, that says Plain Text by default to change this, OR use the command menu (Ctrl+Shift+P), choose Change Language Mode (should be right at the top, usually. If not, simply type "lang"), confirm with Enter, choose JSON, confirm with Enter again, et voilà, you are ready to go!
    • VSCode in JSON mode will automatically detect all syntax issues and problems, making the creation of a valid JSON config file very easy!
• Alternatively, use the more complicated way and check your config with online tools to validate a JSON text: Too many to list all, probably. But some examples:
  • https://jsonlint.com/
  • https://www.jslint.com/

Examples to demonstrate the layout of a config file

Empty config file, just showing the different sections (the top level sections)

{
    "extractor":
    {

    },

    "downloader":
    {

    },

    "output":
    {

    },

    "postprocessor":
    {

    }

}

All Sections again, including some options as an example

{
    "extractor":
    {
        "base-directory": "~/Downloads/All",
        "archive": "~/gallery-dl/archive/gallery-dl_archive_global.sqlite3",

        "keywords": {"bkey": "", "ckey": ""},
        "keywords-default": "",

        "path-restrict": {
            "\\": "⧹",
            "/" : "⧸",
            "|" : "￨",
            ":" : "꞉",
            "*" : "∗",
            "?" : "？",
            "\"": "″",
            "<" : "﹤",
            ">" : "﹥"
        },

        "postprocessors": [
            {
                "name": "metadata",
                "mode": "tags",
                "whitelist": ["danbooru", "moebooru", "sankaku"]
            }
        ],

        "#_comment_1": "All settings for an extractor belong here!",
        "#_comment_2": "Site-specific! Set in their own object, like this, for example:",
        "twitter":
        {
            "user":
            {
                "directory": {
                    "retweet_id": ["Twitter", "Timelines", "{user[name]}", "RT"],
                    "quote_id"  : ["Twitter", "Timelines", "{user[name]}", "QT"],
                    ""          : ["Twitter", "Timelines", "{user[name]}"]
                },
                "filename": "{author[name]!t:R.//}{date:%Y-%m-%d}.{tweet_id}.{filename!t:R.//}{count:?.//>04}.{extension}"
            },

            "tweet":
            {
                "directory": {
                    "locals().get('bkey')": ["Twitter", "Tweets", "{bkey}"],
                    ""                    : ["Twitter", "Tweets", "Unsorted"]
                },
                "filename": "{author[name]}{date:%Y-%m-%d}.{tweet_id}.{filename}{count:?.//>04}.{extension}",
                "conversations": true,
                "replies": "self",
                "expand": true,
                "postprocessors": ["modtime", "twitter-thread"]
            },

            "archive-prefix": "",
            "archive": "~/gallery-dl/archive/gallery-dl_archive_twitter.db",

            "include": ["timeline", "media"],
            "postprocessors": ["nonpicsort"],

            "sleep": [0.2, 0.8],
            "sleep-request": [0.2, 0.8],

            "cookies": {
                "auth_token": "<your token here>"
            }
        },

        "ytdl":
        {
            "module": "yt_dlp",
            "forward-cookies": true,
            "raw-options": {
                "no_color": true,
                "verbose": false,
                "quiet": false
            },
            "format": "bv*+ba/b"
        }
    },

    "downloader":
    {
        "progress": 32.0,
        "retries": 12,
        "timeout": 32.0,
        "part": true,

        "http":
        {
            "rate": null,
            "verify": true
        },

        "ytdl":
        {
            "module": "yt_dlp",
            "forward-cookies": true,
            "logging": true
        }

    },

    "output":
    {
        "mode": "color",
        "ansi": true,
        "progress": true,

        "log": {
            "level": "info",
            "format-date": "%Y-%m-%dT%H:%M:%S",
            "format": {
                "debug"  : "\u001b[0;37mDebug  :  {name} -> {message}\u001b[0m",
                "info"   : "\u001b[1;37mInfo   :  {name} -> {message}\u001b[0m",
                "warning": "\u001b[1;33mWarning:  {name} -> {message} {extractor.url:?[/]/}\u001b[0m",
                "error"  : "\u001b[1;31mError  :  {name} -> {message} {extractor.url:?[/]/}\u001b[0m"
            }
        },

        "logfile": {
            "path": "~/gallery-dl/logs/gallery-dl.log.txt",
            "mode": "a",
            "format": {
                "debug"  : "[{asctime}][{levelname}] {message}",
                "info"   : "[{asctime}][{levelname}] {message}",
                "warning": "[{asctime}][{levelname}] {message} [Source URL: {extractor.url}]",
                "error"  : "[{asctime}][{levelname}] {message} [Source URL: {extractor.url}]"
            },
            "format-date": "%Y-%m-%dT%H:%M:%S",
            "level": "info"
        },

        "unsupportedfile": {
            "path": "~/gallery-dl/logs/gallery-dl.unsupported.txt",
            "mode": "a",
            "format": "[{asctime}] {message}",
            "format-date": "%Y-%m-%dT%H:%M:%S"
        }

    },

    "postprocessor":
    {
        "nonpicsort":
        {
            "name": "classify",
            "mapping":
            {
                "Clips"    : ["mp4", "mkv", "webm", "flv", "ogv", "wmv", "avi", "mpg", "mpeg", "3gp", "vob", "ts"],
                "Archives" : ["zip", "rar", "7z", "tar", "gz", "bz2", "ace", "xz", "lz"]
            }
        },

        "modtime":
        {
            "name": "mtime",
            "event": "post"
        },

        "twitter-thread":
        {
            "name": "metadata",
            "event": "post",
            "mtime": true,
            "mode": "custom",
            "content-format": "\fT ~/gallery-dl/config/twitter-thread.html",
            "filename": "Twitter__{author[name]}__{date:%Y.%m.%d}__{retweet_id|tweet_id}.html",
            "directory": "Thread",
            "archive-format": "{tweet_id}_{retweet_id}_{num}",
            "archive-prefix": "",
            "archive": "~/gallery-dl/archive/gallery-dl_archive_twitter_metadata.db"
        }
    }

}

Explanations for the options listed above:

• "base-directory"
• "archive"
• "keywords"
• "keywords-default"
• "path-restrict"
• ... [wip]

What the example above is intended to demonstrate:

• Using config settings in the top level sections "extractor", "downloader", "output", "postprocessor"
• Options for both the ytdl extractor and the ytdl downloader - which is not the same thing
  • The difference, in a gist, is basically just:
    • Site is not supported by gallery-dl itself : Uses extractor.ytdl
    • Site is supported by gallery-dl itself       : Uses downloader.ytdl
• The three different locations to set up a post-processor
  1. JSON array ("postprocessors": [ {...}, {...} ]) directly at the Base level: "extractor.postprocessors: [ {...}, {...} ]"
     • Always used, but respecting post-processor options like "whitelist" and "blacklist"
  2. Site-specific post-processor, JSON array ("postprocessors": [ {...}, {...} ]) at the Category level: "extractor.twitter.postprocessors: [ {...}, {...} ]"
  3. The post-processor block ("postprocessor": { }) at the Top level: Note that you have to bind every post-processor object herein to a custom name.
     • Every post-processor can then be referenced by their name in the JSON array at the Category level (The example for "twitter" above shows this by referencing three different post-processor configurations)
• Using a permanent logfile and unsupported URL file with some customizations

Options of gallery-dl can be specified on multiple levels of the configuration tree:

Base level:	extractor.<option-name>
Category level:	extractor.<category>.<option-name>
Subcategory level:	extractor.<category>.<subcategory>.<option-name>

---