Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

alexklwong · 2020-03-25T08:42:11Z

I want to download a 16GB zip file on Google Drive using:

gdown https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8

I have already set the permissions on the file to ``Anyone with the link''

But I get this error:

Permission denied: https://drive.google.com/uc?id=1InfIal4y7OBMGNUDeldEmDxtD0MrewY8
Maybe you need to change permission over 'Anyone with the link'?

mrlzla · 2020-03-25T12:16:30Z

The same error

zenithfang · 2020-03-26T00:35:36Z

same error

wkentaro · 2020-03-31T06:08:11Z

Maybe the same issue as #26.

I found this happens with large data + many access (e.g., public dataset) #42. However, I have no idea how to fix this.

If they are the same issue, there's no solution at the moment from gdown side.

Rmao99 · 2020-03-31T06:12:48Z

I have this issue as well with a 15GB dataset and a 4.3 GB dataset. But when I copy the 4.3GB to my local drive and manually get the link on my copied dataset it works. Can't make a copy of the 15GB on free drive though.

For reference the datasets im downloading are:
https://drive.google.com/file/d/1NrqOLbMa_RwHbG3KIXJFWLnlND2kiIpj/view
https://drive.google.com/file/d/15w0q3Nuye2ieu_aUNdTS_FNvoVzM4RMF/view

I was actually able to successfully download both datasets once from the original links but after needing to redo the downloads they no longer work even when uninstalling and reinstalling gdown

wkentaro · 2020-03-31T06:18:31Z

@Rmao99 Thanks for the information. That behavior makes sense if that's the issue of Google-Drive's side. It seems they restrict access from the command-line for data with many access (e.g., public dataset).

wkentaro · 2020-04-29T17:33:01Z

I've added a new feature to keep cookies #51.
I hope this fixes this issue.

You can try pre-released version of this.

pip install -U --no-cache-dir gdown --pre

by default, cookies is enabled and saved into ~/.cache/gdown/cookies.json.
You can disable it by gdown --no-cookies XXX

Jackie-LJQ · 2020-05-05T00:41:40Z

I have the same issue. I tried the above version, not fix.

whulixiya · 2020-06-01T03:17:46Z

I met the same error when I try to download many big files from my google drive, have anyone fixed it?

carlmalamud · 2020-06-03T14:23:24Z

I find if I rm ~/.cache/gdown/cookies.json and then restart my script, things start downloading again. Is your cookie expiring perhaps? I'm good for 1,000 or so files before I get the error.

Thanks for gdown. Very helpful!

wkentaro · 2020-06-03T16:28:32Z

@carlmalamud Thanks for info. Do you have any sharable links to reproduce this?

whulixiya · 2020-06-04T00:34:45Z

Unfortunately, there is nothing in ~/.cache/gdown/

…

On Wed, Jun 3, 2020 at 10:23 PM Carl Malamud ***@***.***> wrote: I find if I rm ~/.cache/gdown/cookies.json and then restart my script, things start downloading again. Is your cookie expiring perhaps? I'm good for 1,000 or so files before I get the error. Thanks for gdown. Very helpful! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#43 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJDECDYFMKEGB3BS3FBLVETRUZMGZANCNFSM4LTI65IQ> .

-- *Xiya Li* *State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, Wuhan University, Wuhan, China* *(M**aster**)*

carlmalamud · 2020-06-04T18:32:29Z

@wkentaro What is happening is your cookie keeps growing and growing until it is full. I believe every time there is a warning, a new line gets added. I've seen my cookie grow from 11 warnings to 36 warnings in about 2 hours. After about 1,000 files, there are presumably enough that there isn't room for more warnings, so you when present the cookie that doesn't mention a particular download, Google rejects it and your code is thinking that it is a permission issue.

Perhaps if your cookie fails, you could remove it, try again, and only then if it fails a second time assume it is a permissions issue.

Here is a snippet of what my cookie looks like:

[
[
"AUTH_22nmp4ombsvnvmjrl3nb3u3hsrucc95e",
"14651752893030988712Z|1591294950000|hospjjjbfh4rapv59tff247rc9fhcbr3"
],
[
"NID",
"204=x4h9LoiFMtrKRVT3aYaAXNxcePhv5SjsTw9qrioaMhrWD61BLoO_hbjosmcIBEYKSEf-pqz0U5D9SyUKwCiHJJ1Ys_xGBVg8oLqizpJaVYdWUoSIgSZKNL3xpkoGvIQs9lz8hTQD9EgvcwunJ--j5FtM7OPAn_W8mnVaHEVweSs"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEMExkMlBpOUJoQ28",
"IcVe"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEMmVDYVJUckNQZlE",
"Ricd"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENGlEZDNabFRwVzg",
"579P"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENk9JWmNMNm81R0k",
"o4Bf"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQENlVqTnJsc2R5dzA",
"osl-"
],
[
"download_warning_13058876669334088843_0B7JhzNLs-FQEOGpkZzFad25SdFE",
"8XbE"
],

carlmalamud · 2020-06-06T21:15:25Z

It looks like the magic number is 110 download_warnings and the cookie is full.

wkentaro · 2020-06-06T21:22:48Z

@carlmalamud thanks!

wkentaro · 2020-06-06T21:28:48Z

Sent a PR here: #55

wkentaro · 2020-06-06T21:32:50Z

Released as v3.11.1

wkentaro · 2020-06-06T21:33:35Z

@carlmalamud can you try if the new version eases the issue?

grudloff · 2020-06-10T18:26:03Z

Same problem, upgraded to latest release and I get the following message:

Access denied with the following error:

 	Too many users have viewed or downloaded this file recently. Please
	try accessing the file again later. If the file you are trying to
	access is particularly large or is shared with many people, it may
	take up to 24 hours to be able to view or download the file. If you
	still can't access a file after 24 hours, contact your domain
	administrator. 

You may still be able to access the file from the browser:
%%%URLHERE

carlmalamud · 2020-06-16T20:39:10Z

I was chugging along nicely for a while, but I now am also getting same error as @grudloff. I was in fact able to pull the file up in a browser and start the download as indicated above. I looked in the gdown cookie and didn't see any download_warning lines, so evidently we never got the cookie back from them with the additional authorization.

(I have now installed 3.11.1, but was running on 3.11.0 before, I simply inserted a "rm ~/.cache/gdown/cookies.json" every fifty lines. Sorry, it was working so I didn't want to bother it. :). The upgrade doesn't seem to help on the above error.

carlmalamud · 2020-06-16T20:46:19Z

I am imagining that the error might have something to do with my being an authenticated user in the browser and not via gdown. I'm not sure is presenting an oauth token might make a difference here?

wkentaro · 2020-06-17T00:39:23Z

I'm not sure is presenting an oauth token might make a difference here?

I don't think so, gdown uses requests inside without using any browsers.

kshitijagrwl · 2020-07-06T09:30:01Z

Solved by disabling certificate verification by changing line 111 in download.py

res = sess.get(url, stream=True, verify=False)

However it does display this warning
InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning

wkentaro · 2020-07-06T12:44:40Z

Solved by disabling certificate verification

In my case, it didn't work.

e96031413 · 2024-01-16T01:48:57Z

Older version gdown also helps.

pip install gdown==4.6.0

I finally solved this issue by using 4.6.0 gdown to download files.

aasyedd1 · 2024-01-16T13:04:00Z

@e96031413 worked like a charm. thanks alot

neeeal · 2024-01-17T13:36:40Z

pip install gdown==4.6.0

actually solved my issue. thanks

wkentaro · 2024-01-20T04:05:58Z

This PR #300 should help this issue in some regard. I also added FAQ to the README.
https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-still-cant-download

Releasing v5.0.0 soon.

pmeier · 2024-01-29T08:39:33Z

When hitting large files that GDrive wants extra confirmation before it starts the download, e.g. https://drive.google.com/uc?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK, we currently hit this branch:

gdown/gdown/download.py

Lines 31 to 35 in ae122f4

    
           m = re.search('id="download-form" action="(.+?)"', line) 
        
           if m: 
        
               url = m.groups()[0] 
        
               url = url.replace("&amp;", "&") 
        
               break

The relevant corresponding HTML looks like

<form id="download-form" action="https://drive.usercontent.google.com/download" method="get">
	<input type="submit" id="uc-download-link" class="goog-inline-block jfk-button jfk-button-action" value="Download anyway"/>
	<input type="hidden" name="id" value="1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK">
	<input type="hidden" name="confirm" value="t">
	<input type="hidden" name="uuid" value="ce320bab-e9f0-4d0d-a9d8-8cede0161915">
</form>

The branch above just extracts https://drive.usercontent.google.com/download from it and this later fails other checks as this doesn't provide a file.

What we should do here is to extract the other parts of the form as well and append them as query. So for this example https://drive.usercontent.google.com/download?id=1r6o0pSROcV1_VwT4oSjA2FBUSCWGuxLK&confirm=t&uuid=ce320bab-e9f0-4d0d-a9d8-8cede0161915. As you can see this link also includes confirm=t that multiple people in this thread have already mentioned. I'll send a PR to fix this.

pmeier · 2024-01-29T08:53:30Z

@wkentaro see #308. Could you please have a look?

wkentaro · 2024-02-01T14:05:39Z

Hi everyone.

I've just merged #308 by @pmeier, which adds extra pattern to parse the url to download.

Since I've never seen that pattern, I suspect it is a regional issue (I'm accessing from Japan, @pmeier is accessing from Germany). But at least It should help people who have the same issue as him.

vanloc1808 · 2024-02-23T13:51:21Z

@wkentaro Hi, I observed that only downgrading to 4.6.0 helps, new version does not help :(

wkentaro · 2024-02-23T13:57:43Z

Can you please give me an example url?

vanloc1808 · 2024-02-24T03:54:41Z

https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link
Sure, here is it

vanloc1808 · 2024-02-24T03:56:04Z

I tried both

gdown --fuzzy 'https://drive.google.com/file/d/1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb/view?usp=drive_link'

and

gdown https://drive.google.com/uc?id=1NmGrEJ1uxrrBgf98UCU9SgpgsGEs3sLb

But it only works for me with gdown==4.6.0

wkentaro · 2024-02-25T02:48:58Z

@vanloc1808 It seems working for me with the latest.

Maybe the cookies file is not helping? If so remove it by rm ~/.cache/gdown/cookies.txt

vanloc1808 · 2024-02-25T04:15:15Z

@wkentaro it works for me, thank you!

Junyi42 · 2024-03-02T07:20:39Z

Older version gdown also helps.
pip install gdown==4.6.0
I finally solved this issue by using 4.6.0 gdown to download files.

Restarting the terminal after the downgrading works for me!

AndrewLt · 2024-03-07T16:14:05Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

julienboussard · 2024-03-08T15:16:56Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Yeah this happened to me as well several times (with different versions). I'm not sure why, but changing the version (sometimes to a newer one, sometimes to an older one) seem to work, but always with a different version. Maybe it clears something and allows gdown to download again? I usually try versions 4.6.0, 5.1.0, or the latest (pip install --upgrade gdown). I also remove cookies.txt, and I end up (randomly) with a version that works. Hope this helps

ArianAmani · 2024-03-12T14:39:23Z

I had a funny moment today. I got the same issue explained in the first message on the topic. The solution was to reinstall a several times version of gdown and finally install 5.1.0 ver to solve the problem. But this version had this issue when I started searching for the solution 😆

p.s. it is for colab

Weirdly this worked for me too. And I was using the default gdown (pip install gdown) on colab using gdown --id ID without a problem until just a few days ago. Thanks!

ain-soph · 2024-06-14T05:01:35Z

@wkentaro I'm still suffering this issue today. (gdown version 5.2.0)

I'm trying to download my own uploaded dataset (10G)
https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

And it raises the error

gdown https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_


Failed to retrieve file url:

        Too many users have viewed or downloaded this file recently. Please
        try accessing the file again later. If the file you are trying to
        access is particularly large or is shared with many people, it may
        take up to 24 hours to be able to view or download the file. If you
        still can't access a file after 24 hours, contact your domain
        administrator.

You may still be able to access the file from the browser:

        https://drive.google.com/uc?id=1vv73kAHiKb4KiL_oIH4DOWzUoaTeKzt_

but Gdown can't. Please check connections and **permissions.**

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

wkentaro · 2024-06-14T11:10:17Z

@ain-soph It works for me now, so looks like too many requests?

@JiaHeng-DLUT 's workaround works for me (https://stackoverflow.com/questions/65312867/how-to-download-large-file-from-google-drive-from-terminal-gdown-doesnt-work)

We can't do this because the scope of gdown is without logging in.

this is another workaround: https://github.com/wkentaro/gdown?tab=readme-ov-file#i-set-the-permission-anyone-with-link-but-i-still-cant-download

wkentaro self-assigned this Mar 31, 2020

wkentaro added more-information-needed labels Mar 31, 2020

wkentaro mentioned this issue Apr 29, 2020

Add cookies saved in home cache directory #51

Merged

wkentaro mentioned this issue May 14, 2020

Permission denied after few donwload #53

Closed

chi0tzp mentioned this issue May 15, 2020

[R2V2 dataset download] Permission denied - Maybe you need to change permission over 'Anyone with the link'? danielgordon10/vince#1

Closed

wkentaro mentioned this issue Jun 6, 2020

Remove download_warning_ from cookies #55

Merged

This was referenced Jan 10, 2024

FileNotFoundError in test_perceptual_loss Project-MONAI/MONAI#7382

Closed

Missing file in FileNotFoundError in profiling_camelyon_pipeline.ipynb Project-MONAI/tutorials#1608

Closed

biochunan mentioned this issue Jan 10, 2024

gdown failed downloading assets biochunan/CDRConformationClassification#5

Closed

faridyagubbayli mentioned this issue Jan 11, 2024

[BUG] Phantom data cannot be downloaded from google drive waltsims/k-wave-python#250

Closed

shivammehta25 mentioned this issue Jan 12, 2024

Matcha-TTS access denied shivammehta25/Matcha-TTS#41

Closed

maltelueken mentioned this issue Jan 17, 2024

Prepare release 1.0 mexca/mexca#111

Merged

wkentaro unpinned this issue Jan 20, 2024

pmeier mentioned this issue Jan 29, 2024

Add extra pattern to extract url from download-form returned by Google Drive for a large file #308

Merged

wkentaro closed this as completed in #308 Feb 1, 2024

wkentaro reopened this Feb 1, 2024

Jieran-S mentioned this issue Mar 18, 2024

Update dataset gdown version SpatialHackathon/SpaceHack2023#249

Merged

Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

Permission denied ... Maybe you need to change permission over 'Anyone with the link'? #43

Comments

alexklwong commented Mar 25, 2020

mrlzla commented Mar 25, 2020

zenithfang commented Mar 26, 2020

wkentaro commented Mar 31, 2020 • edited Loading

Rmao99 commented Mar 31, 2020

wkentaro commented Mar 31, 2020 • edited Loading

wkentaro commented Apr 29, 2020

Jackie-LJQ commented May 5, 2020 • edited Loading

whulixiya commented Jun 1, 2020

carlmalamud commented Jun 3, 2020 • edited Loading

wkentaro commented Jun 3, 2020

whulixiya commented Jun 4, 2020 via email

carlmalamud commented Jun 4, 2020

carlmalamud commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

wkentaro commented Jun 6, 2020

grudloff commented Jun 10, 2020

carlmalamud commented Jun 16, 2020

carlmalamud commented Jun 16, 2020

wkentaro commented Jun 17, 2020

kshitijagrwl commented Jul 6, 2020 • edited Loading

wkentaro commented Jul 6, 2020

e96031413 commented Jan 16, 2024

aasyedd1 commented Jan 16, 2024

neeeal commented Jan 17, 2024

wkentaro commented Jan 20, 2024

pmeier commented Jan 29, 2024

pmeier commented Jan 29, 2024

wkentaro commented Feb 1, 2024

vanloc1808 commented Feb 23, 2024

wkentaro commented Feb 23, 2024

vanloc1808 commented Feb 24, 2024

vanloc1808 commented Feb 24, 2024 • edited Loading

wkentaro commented Feb 25, 2024

vanloc1808 commented Feb 25, 2024

Junyi42 commented Mar 2, 2024 • edited Loading

AndrewLt commented Mar 7, 2024 • edited Loading

julienboussard commented Mar 8, 2024 • edited Loading

ArianAmani commented Mar 12, 2024

ain-soph commented Jun 14, 2024

wkentaro commented Jun 14, 2024 • edited Loading

wkentaro commented Mar 31, 2020 •

edited

Loading

wkentaro commented Mar 31, 2020 •

edited

Loading

Jackie-LJQ commented May 5, 2020 •

edited

Loading

carlmalamud commented Jun 3, 2020 •

edited

Loading

kshitijagrwl commented Jul 6, 2020 •

edited

Loading

vanloc1808 commented Feb 24, 2024 •

edited

Loading

Junyi42 commented Mar 2, 2024 •

edited

Loading

AndrewLt commented Mar 7, 2024 •

edited

Loading

julienboussard commented Mar 8, 2024 •

edited

Loading

wkentaro commented Jun 14, 2024 •

edited

Loading