danbooru.donmai.us Error 403 Forbidden

[Zuescho]

Does anyone have an idea why danbooru could be throwing this error?
I tried adding my login data and / or cookie, but it always gives me an 403

.\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatyann)"

.\gallery-dl.exe : [danbooru][error] HttpError: '403 Forbidden' for 'https://danbooru.donmai.us/posts.json'
At line:1 char:1
+ .\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatya ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([danbooru][erro....us/posts.json':String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

[mikf]

I tried adding my login data

That's most likely the problem. The password value for Danbooru needs to be the API key found in your profile page and not your actual password.

[Zuescho]

Sadly it is not working even without any login, just tried using the API Key and still get the same error, I can view the post.json in browser but that's it

PS C:\CommandLine\gallery-dl> .\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatyann)" -v
.\gallery-dl.exe : [gallery-dl][debug] Version 1.23.5 - Executable
At line:1 char:1
+ .\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatya ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([gallery-dl][de....5 - Executable:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.28.1 - urllib3 1.26.12
[gallery-dl][debug] Starting DownloadJob for 'https://danbooru.donmai.us/posts?tags=rouka_(akatyann)'
[danbooru][debug] Using HTTP Basic Auth for user '*Deleted*'
[danbooru][debug] Using DanbooruTagExtractor for 'https://danbooru.donmai.us/posts?tags=rouka_(akatyann)'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=rouka_%28akatyann%29&limit=200 HTTP/1.1" 403 None
[danbooru][error] HttpError: '403 Forbidden' for 'https://danbooru.donmai.us/posts.json'

[mikf]

even without any login
...
[danbooru][debug] Using HTTP Basic Auth for user 'Deleted'

You still have a username set somewhere, otherwise HTTP Basic Auth would not get used.
Try with -o username=.

edit: I can't read ...

[mikf]

Never mind, you'd get 401 Unauthorized for wrong username/password, not a 403.
What does --write-pages output show?

[Zuescho]

Sorry for the confusion, I just tried to hide there.
I managed to get it working now by supplying my whole cookies.txt

So if maybe anyone else has this problem, try

"danbooru":
        {
            "username": null,
            "password": null,
            "ugoira": false,
            "cookies": "Path/To/cookies.txt"

On Windows, you need to replace backward slashes ( \ ) to forward ones ( / )

Also, just in case, I set the user-agent to my real user-agent (in my case)
"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0",

What does --write-pages output show?

PS C:\CommandLine\gallery-dl> .\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatyann)" -v --write-pages
.\gallery-dl.exe : [gallery-dl][debug] Version 1.23.5 - Executable
At line:1 char:1
+ .\gallery-dl.exe "https://danbooru.donmai.us/posts?tags=rouka_(akatya ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: ([gallery-dl][de....5 - Executable:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
[gallery-dl][debug] Python 3.7.9 - Windows-10-10.0.19041
[gallery-dl][debug] requests 2.28.1 - urllib3 1.26.12
[gallery-dl][debug] Starting DownloadJob for 'https://danbooru.donmai.us/posts?tags=rouka_(akatyann)'
[danbooru][debug] Using DanbooruTagExtractor for 'https://danbooru.donmai.us/posts?tags=rouka_(akatyann)'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): danbooru.donmai.us:443
[urllib3.connectionpool][debug] https://danbooru.donmai.us:443 "GET /posts.json?tags=rouka_%28akatyann%29&limit=200 HTTP/1.1" 403 None
[danbooru][error] HttpError: '403 Forbidden' for 'https://danbooru.donmai.us/posts.json'

[mikf]

--write-pages writes the response of any HTTP request to a file on disk in your current working directory, including the 403 error. Maybe that has some more details.

[Zuescho]

<!DOCTYPE html>
<html lang="en-US">
<head>
    <title>Just a moment...</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">
    <meta name="robots" content="noindex,nofollow">
    <meta name="viewport" content="width=device-width,initial-scale=1">
    <link href="/cdn-cgi/styles/challenges.css" rel="stylesheet">
    

</head>
<body class="no-js">
    <div class="main-wrapper" role="main">
    <div class="main-content">
        <h1 class="zone-name-title h1">
            <img class="heading-favicon" src="/favicon.ico"
                 onerror="this.onerror=null;this.parentNode.removeChild(this)">
            danbooru.donmai.us
        </h1>
        <h2 class="h2" id="challenge-running">
            Checking if the site connection is secure
        </h2>
        <noscript>
            <div id="challenge-error-title">
                <div class="h2">
                    <span class="icon-wrapper">
                        <div class="heading-icon warning-icon"></div>
                    </span>
                    <span id="challenge-error-text">
                        Enable JavaScript and cookies to continue
                    </span>
                </div>
            </div>
        </noscript>
        <div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=762d0d9fefaa5b50')"></div>
        <div id="challenge-body-text" class="core-msg spacer">
            danbooru.donmai.us needs to review the security of your connection before proceeding.
        </div>
        <form id="challenge-form" action="/posts.json?tags=rouka_%28akatyann%29&amp;limit=200&amp;__cf_chl_f_tk=vy4KDqGhOTTPo1.uYW.zLKOXff1lbZuCFh3F9rcx0.8-1667226255-0-gaNycGzNB70" method="POST" enctype="application/x-www-form-urlencoded">
            <input type="hidden" name="md" value="Z_YEFuQ7fu8mDMLU4ypAMSSX1lBek5oafP4BdYf6YL4-1667226255-0-ATbvUydSaWEDUFUVXcVpe012MOC-xhbc9laW_Kua4TjOU3oNWN7NZwnpROMA8juHTW-4KGE-oN8GFZIsjCFKb7QKSnV-DCPjPt5YwTM_qrp1yksuCmHKENJMqvEYNMMPrv4gG_c644TRCMEvhFpxQLWECy0VZgTUce8b30rNkazBwmOhEVcy00NB2cTppvJLMsa9eouDo6e4mXV12bbgeMXOiAIoDzVzSlb3UQnoJK4gSAAzdyJ8N7LY81JCgjbS2Qbthxv5A-XaGLvD6GfFiqcRsbDAIVn3JqcrReUmA6JqUqpGKPpTSyGeHOjP4H66Snqg0vjItV-tyjYQ_al7iP0FRhv6P6ejnMJwPyRzPd18fWyQ-Dd24ALGW5o_f9cpdZeQg7Vrt7SpnS_MPo4DsoQD0xfebskDYLMJZkG5kfxC07YTnPkmDhVAnHpI4L2hpI3dRyiW5VIJc1bUqrlZjiSlSX0DO38QPMyf43dQ55bIlLY1s2OthcncoZcVxvv-ud5joV6Ujt7tTsW_v1xloMqKVN9TfZowjRMRlvzwAhG7-oBaFGn8kd64KHn7B4bEKFDZDpPz9_j6cMiYn790_F6ydDClhaVZab8KFBLctT_y4qlCsszT8woZ3oMPmv36gKRXMIoDd86i5kMat6q1y-HJEz0WYxhKc_0v7lmDQxgUY9lOifCPeb6ON57EyTSHXLkQbHDjH3B8JxHdUdCzAUEMv-8q6TNzHwKph36VrMFo1igc1PUwWxFEbmTwhga5A2GZjOqbxPxjwB_Tm_qmLmg">
            <input type="hidden" name="r" value="NGCjsbcSH2a1O6wX3rikoON1Umgmbia4axs23r4iNrU-1667226255-0-AezJkh6UkoFnCQ4C/D1zioeq1ZBXif4GTHXtudn3O8SgsiRTA9r8NegEP1U1QFpE1Av85lsuboMrTi4Y4fNtWpZ9KFMQ78S4yPPH6PutGkf6SQrilPB7O0Am7jDXZN30oLIBrSJoXFDWohO8OMqxVzCZrCNBV04en1gbZPI6AxCRO4ZWidQLdSIoAdjGvvI/s1oIITHEQWBbnzAz9CnhKMffLGEYodmhjKHu+RtzZ7QXUQOcMDpC2LMIZQFxJ6Vq6S8h0cGZ8gD230KX9aHk8hC2l02hgPm8Z+8e9eSaz7PPBXMmhgYciEXSvZ9C8VOhARbckhN39l0yWs4baXQsS43IZ46bqZV1WE6+WQEd8jU0pJgaZkuWYeSI+lKsS6irBqbWnz8wEDcNdJ6eh60GqJ+9CMwNg9cMG+7niQNXa6D/2IgUCyYW5VeXTEJngVkVjZkfqpeeAhg1HcPWrnDlvGAE/VZsceHm5Z+PvNYuTl8KDUY9dUZL9hYXZSk53rPPlaU1alv3eMhgrG5TBQadmav1hqlhj1/d37c2NBfSS0c10160yJpOpiBDUYfDvbu544kifouerZdwUT/zrjxuiGN3kanBP+igMiSK7YITQZW8HLvi80Q/hQRa7xdgxlvlNPWZ0VWH3m1PaoeM2JQ9dNBB2SoyzO7quKNy0tcP+cmevwWzqSARfdhX+sqArW/PliOeE8XuqD6iHtZKa6toN/mnpi/TTyH4n8WvkTSxYZnw4mhAjICarvGtsqB6kH96d3b+2AFiFl6yIYEDzRkSojDTuxRo/CMNEfS0MchDQ0bZ98R012avOsZjl+4Mp5cbBrLR3llPGRsgWiJphGw8L0aghhbgPExq5vmqEUWDHNluOmiBDePK/yMuRwYwF5gL1GSgspYyuGkLt/tYUYQVZy0GT5nI8OdQZdqvWy7rUIhmI6SDIhuHpX91IAUhKj7+s5C+RwSnAKqzpAWLLYc0ALTSZW/clz/rxKx15Vy0uLFBzhrrZMfVnut12c1trpi0QLr88+7h07rPazxfVw5o8xiZQGiMOV//ez4pRjz0s2QCcTQ8jBTgSP6B08ycubEMKVP7jMZECv5ZIItzxThYyynna9bCZkn0AuS6x1nX6z7Z8VpsGi28K4hPlbdOl/WEzUL0vcZi/DmYEYC9IkoclYVCCIBkQ4ugLcnTZTRe2hHi00a0hnG1Cl7ddPfSf/y5KMtsSHJt8GMPsFAMCh6CGUUzLs1qIJtFTwgvruLCP1eBY54c0N6TzVjF9es40JgCtDJgg9ZSFWhIREsl5vENnMVBZfp1wMK7JvyGGQGF/Lnkcch8PeROUxwm4LwAeALBmbT/fDPpLb+BBVSRh8x68/zSJcn4NGvQv72t5l4J1iPDDe3X3Sand6iJyC52VFpj2BO/AygTwlKG5+cMVgEMUWBcLgBgC84zwOTdEIqmRjCPGd1ZvV1qkBysVQ9UTSTkqsQBKnzWxao0iSX6CyaprCmBndQ3JxHxFhhZfQQQ/zEGh+6Ib08YePKiJ4PJW6JS2LAde977bpsjBmIjE34RgnxmePguUFlg4ciwzCXtqniLwwWhZnp7yhy6CgL15ZoWhvql6tsNzSBTyW1MpaTSWIx4q38m90j5uEf9kHd1aPAFF/bxRMHR3+pDFXIQI9Yogyai3GYO1pTlOjQZkr4lm2vDjQ6ItaJpHRPWGiQ4HDY8JRK/MOSeK9XxPcO4+osd8Oib8opWljn2nsmEQJffvPy04u222jvV4kwFBRdg6j6J5Sn9BEMupp2wbjTqUFDtsfD9wjXWSSXfVye9RrF2qCmY8BN4Ar0vErw88B23VUqqqNrV86kWmmJUj7Q6cbD7Fd89Y+n4bZ7NkslRpJ9I8w+kCEaDhDlog2JXefR3wqcee6hnz/E35HSThh7AVqR9Ug==">
        </form>
    </div>
</div>
<script>
    (function(){
        window._cf_chl_opt={
            cvId: '2',
            cType: 'managed',
            cNounce: '58209',
            cRay: '762d0d9fefaa5b50',
            cHash: '6645de5cf5fad6a',
            cUPMDTk: "\/posts.json?tags=rouka_%28akatyann%29&limit=200&__cf_chl_tk=vy4KDqGhOTTPo1.uYW.zLKOXff1lbZuCFh3F9rcx0.8-1667226255-0-gaNycGzNB70",
            cFPWv: 'b',
            cTTimeMs: '1000',
            cTplV: 4,
            cTplB: 'cf',
            cRq: {
                ru: 'aHR0cHM6Ly9kYW5ib29ydS5kb25tYWkudXMvcG9zdHMuanNvbj90YWdzPXJvdWthXyUyOGFrYXR5YW5uJTI5JmxpbWl0PTIwMA==',
                ra: 'TW96aWxsYS81LjAgKFdpbmRvd3MgTlQgMTAuMDsgV2luNjQ7IHg2NDsgcnY6OTkuMCkgR2Vja28vMjAxMDAxMDEgRmlyZWZveC85OS4w',
                rm: 'R0VU',
                d: 'bPjwk3gszDEiGFdXfwjNDIz0nHnD6yzPYwKQJd0ZA2dzoKY1koNmFIFn5xefrbgDA9h8APx3CkqMNJ9MG5HfV5dogj9gjQ5VCtzySFuB6LEhZzRcF/p88bXcrOAyjFa18tCSLZADcXRtgxtNBY+0I3lSFxFi8/A6zgRy8O00E3v5y3mrneRVAuZKHmgn+hGhYOXfxGakhFRmBnb6Jx598cOhUnencIbCxuja7smaMzS7oZjiNnMV5jDOEcYq64HpqHXOsZ9T62nf8Qku8xyXNGGLTlv9mYDQwPXV/H9aXzF9fJQmtscqS5o/7lG7RihC4NxbhIEqMEFCf382jp4Ps0ThjRedoKo2GQBTAYGjr35B1eNQeXpPPf5r1WMFpp9MDXyyv//ol91t4kw7b9ClKEPfkiDRxc+Jfa5C072Bl9lrUiVS1+Y2H0OF60PeZDySmxzGRpmgL+5vxPJBcSYqExBLy8eSbc34O3vPg8McXiw6mrB6aClz+HY5tWv+vJxBHVCZtraH6hw/5wjbVGaJA2I5lxiUqgslKECCHdGdAQ23FhPJchmWck/TyjdtmP/QJ/lIhaGim8VD5Hpov87Bwjaq+0l65b1GoZu24miMejKOW6Bd71CKsvdK/GZ3QEVKj2zXfMxHX7SiMgVRWBhbuE2DhYbI+06dqf7j55T3c10=',
                t: 'MTY2NzIyNjI1NS4zNTUwMDA=',
                m: 'LWo7NSifvDoous1p8YmUfN65iE3CMrP5WWgStAzqtCU=',
                i1: 'rf5unBdpVkflTnZGzziXjg==',
                i2: 'lMyOavEc1kKRfPi6rxSCng==',
                zh: 'UJtMg9WjT3VR50Irsc/LfB/GHnC3R5gY3NRtSQ0M9sA=',
                uh: 'rAryPkN3PSyyoCjp+ddd1ehUgJk5jZEeG0zlkXfa49c=',
                hh: '6zrvWg9WpM8Rgu9ZTNVG7mwYikv+6/NT8xnb5x87J7I=',
            }
        };
        var trkjs = document.createElement('img');
        trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=762d0d9fefaa5b50');
        trkjs.setAttribute('style', 'display: none');
        document.body.appendChild(trkjs);
        var cpo = document.createElement('script');
        cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/managed/v1?ray=762d0d9fefaa5b50';
        window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
        window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
        if (window.history && window.history.replaceState) {
            var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
            history.replaceState(null, null, "\/posts.json?tags=rouka_%28akatyann%29&limit=200&__cf_chl_rt_tk=vy4KDqGhOTTPo1.uYW.zLKOXff1lbZuCFh3F9rcx0.8-1667226255-0-gaNycGzNB70" + window._cf_chl_opt.cOgUHash);
            cpo.onload = function() {
                history.replaceState(null, null, ogU);
            };
        }
        document.getElementsByTagName('head')[0].appendChild(cpo);
    }());
</script>


    <div class="footer" role="contentinfo">
        <div class="footer-inner">
            <div class="clearfix diagnostic-wrapper">
                <div class="ray-id">Ray ID: <code>762d0d9fefaa5b50</code></div>
            </div>
            <div class="text-center">Performance &amp; security by <a rel="noopener noreferrer" href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" target="_blank">Cloudflare</a></div>
        </div>
    </div>
</body>
</html>

[mikf]

That's a Cloudflare IUAM check. Good thing you also changed your user agent string, otherwise just using exported cookies would not have worked.

[Zuescho]

Yeah, I noticed that Cloudflare got a lot more aggressive, for some other tools I need to use https://github.com/FlareSolverr/FlareSolverr to get through but till now had luck on danbooru

[mikf]

I've updated the check for such Cloudflare pages, so it'll show a proper warning now. (86790da)

[ClosedPort22]

Given that the site provides API access (https://danbooru.donmai.us/wiki_pages/help%3Aapi), it doesn't look like they want to outright ban bots from their site, so the IUAM challenge may go away after a while. Or you could reach out to the admins to let them know that their current Cloudflare settings essentially render the API useless.

[mikf]

Maybe Danbooru just got overwhelmed by people grabbing training data for their neural nets, which seems to be rather popular at the moment with Stable Diffusion and such, and Cloudflare did its thing to prevent the worst?

[ClosedPort22]

Maybe Danbooru just got overwhelmed by people grabbing training data for their neural nets, which seems to be rather popular at the moment with Stable Diffusion and such, and Cloudflare did its thing to prevent the worst?

That's possible. I think the challenge can only be turned on manually, although it seems that Cloudflare can decide which level to use. Hopefully it's just a temporary thing.