OSError: [Errno 17] File exists

[tseemann]

Been getting this a bit when I restart jobs that failed due to unreachable network:

NFO: Starting new HTTPS connection (1): ftp.ncbi.nlm.nih.gov
DEBUG: "GET /genomes/all/GCA_001443705.1_ASM144370v1/GCA_001443705.1_ASM144370v1_genomic.gbff.gz HTTP/1.1" 200 185238057
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/bin/ncbi-genome-download", line 11, in <module>
    sys.exit(main())
  File "/home/linuxbrew/.linuxbrew/Cellar/python/2.7.12_1/lib/python2.7/site-packages/ncbi_genome_download/__main__.py", line 78, in main
INFO: Starting new HTTPS connection (1): ftp.ncbi.nlm.nih.gov
    ret = ncbi_genome_download.download(args)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/2.7.12_1/lib/python2.7/site-packages/ncbi_genome_download/core.py", line 60, in download
    args.taxid, args.human_readable, args.parallel)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/2.7.12_1/lib/python2.7/site-packages/ncbi_genome_download/core.py", line 94, in _download
    pool.map(worker, download_jobs)
  File "/home/linuxbrew/.linuxbrew/Cellar/python/2.7.12_1/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/home/linuxbrew/.linuxbrew/Cellar/python/2.7.12_1/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
OSError: [Errno 17] File exists

[kblin]

Hm, I wonder if that's while creating the link. Is this a call with --human-readable?

[tseemann]

Yes, I am using ncbi-genome-download --debug -v -o fungi -H -p 1 -s genbank -r 2 -F genbank fungi

[kblin]

I think this is the link creation logic for --human-readable, that probably needs to check if the link already exists.

[anphung]

@tseemann Would help if you provide ncbi-genome-download version as well as how to reproduce the bug.
@kblin Haven't we already passed all errno.EEXIST?

[kblin]

This still exists in latest master. The problem is that we're not checking if the symlink exists already in
download_and_check. See https://github.com/kblin/ncbi-genome-download/blob/master/ncbi_genome_download/core.py#L289

[anphung]

I see. So we should overwrite it with the new source?

[kblin]

We could check if it points at the same target as the one we want to create, and leave the link alone if it does or remove and recreate otherwise.

[anphung]

Since we would link it to the same target whether it points to the target or not, how about we just unlink and link it to the target in both case?

[kblin]

I guess that would work as well.

[anphung]

Thanks, please have a look at the PR.

[kblin]

Reopening issues that aren't shipped in a release yet.

[kblin]

Shipped in 0.2.3 release