"Login incorrect" when logging to the Broad bundle stops installation

[lbeltrame]

wget --continue --no-check-certificate -O dbsnp_138.hg19.vcf.gz 'ftp://gsapubftp-anonymous:@ftp.broadinstitute.org/bundle/2.8/hg19/dbsnp_138.hg19.vcf.gz'
--2015-10-20 12:46:32--  ftp://gsapubftp-anonymous:*password*@ftp.broadinstitute.org/bundle/2.8/hg19/dbsnp_138.hg19.vcf.gz
           => ‘dbsnp_138.hg19.vcf.gz’
Resolving ftp.broadinstitute.org (ftp.broadinstitute.org)... 69.173.80.251
Connecting to ftp.broadinstitute.org (ftp.broadinstitute.org)|69.173.80.251|:21... connected.
Logging in as gsapubftp-anonymous .
Login incorrect.

I assume it's due to limitations in anonymous accesses, but the result is that, given it's a login incorrect prompt, the installation fails because wget exits.

[chapmanb]

Luca;
It looks like the Broad servers are extra unhappy recently, see the discussion from yesterday (#1066). You could bump the issue with the GATK folks:

http://gatkforums.broadinstitute.org/discussion/1215/how-can-i-access-the-gsa-public-ftp-server

but from the bcbio side I don't know of a good mirror so our only workaround is to re-try. Install/upgrade should pick off where it left off.

[lbeltrame]

I would, if the gatkforums didn't have issues with the ISP serving the institution. ;) I wonder if it's a region specific issue, I'll test from another machine just in case.

[lbeltrame]

I noticed that the installer always creates a new folder where to download the bundle. Wouldn't it be better to re-use the existing folder so that one could put the file even if downloaded from another source to allow the install to continue.

[chapmanb]

Luca;
It downloads into a temporary directory then copies the files over. f you want to manually download or copy from another source, place them into the variants directory and it should skip the download on the next run. Hope this helps.

[lfaller]

Hi all,

I am getting a similar error message when trying to install bcbio (see below). It retries a few times and then gives up. Do you know if this is the same error? Has anyone found a workaround? Is there a mirror that has the same data? (In the meantime, I've randomly been trying to rerun the install script, but the results seem to be the same).

Thanks for any suggestions!
~Lina

...
Running GGD recipe: dbsnp   
--2016-02-23 17:22:44--  ftp://gsapubftp-anonymous:*password*@ftp.broadinstitute.org/bundle/2.8/b37/dbsnp_138.b37.vcf.gz
           => `-'
Resolving ftp.broadinstitute.org (ftp.broadinstitute.org)... 69.173.80.251
Connecting to ftp.broadinstitute.org (ftp.broadinstitute.org)|69.173.80.251|:21... connected.
Logging in as gsapubftp-anonymous ...
Error in server response, closing control connection.
Retrying.

... it retries a few times

--2016-02-23 17:22:54--  ftp://gsapubftp-anonymous:*password*@ftp.broadinstitute.org/bundle/2.8/b37/dbsnp_138.b37.vcf.gz
  (try: 5) => `-'
Connecting to ftp.broadinstitute.org (ftp.broadinstitute.org)|69.173.80.251|:21... connected.
Logging in as gsapubftp-anonymous ...
Login incorrect.

gzip: stdin: unexpected end of file
Traceback (most recent call last):
  File "/home/lfaller/local/share/bcbio/anaconda/bin/bcbio_nextgen.py", line 4, in <module>
    __import__('pkg_resources').run_script('bcbio-nextgen==0.9.6', 'bcbio_nextgen.py')
  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 724, in run_script

  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.1.1-py2.7.egg/pkg_resources/__init__.py", line 1650, in run_script

  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.6-py2.7.egg-info/scripts/bcbio_nextgen.py", line 207, in <module>
    install.upgrade_bcbio(kwargs["args"])
  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 89, in upgrade_bcbio
    upgrade_bcbio_data(args, REMOTES)
  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/site-packages/bcbio/install.py", line 257, in upgrade_bcbio_data
    cbl_deploy.deploy(s)
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/deploy/__init__.py", line 65, in deploy
    _setup_vm(options, vm_launcher, actions)
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/deploy/__init__.py", line 110, in _setup_vm
    configure_instance(options, actions)
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/deploy/__init__.py", line 268, in configure_instance
    setup_biodata(options)  
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/deploy/__init__.py", line 250, in setup_biodata
    install_proc(options["genomes"], ["ggd", "s3", "raw"])
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 345, in install_data
    _prep_genomes(env, genomes, genome_indexes, ready_approaches)
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 474, in _prep_genomes
    retrieve_fn(env, manager, gid, idx)
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/genomes.py", line 796, in _install_with_ggd
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 30, in install_recipe
    recipe["recipe"]["full"]["recipe_type"])
  File "/home/lfaller/tmpbcbio-install/cloudbiolinux/cloudbio/biodata/ggd.py", line 62, in _run_recipe
    subprocess.check_output(["bash", run_file])
  File "/home/lfaller/local/share/bcbio/anaconda/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['bash', '/home/lfaller/local/share/bcbio/genomes/Hsapiens/GRCh37/txtmp/ggd-run.sh']' returned non-zero exit status 1
Traceback (most recent call last):
  File "bcbio_nextgen_install.py", line 245, in <module>
    main(parser.parse_args(), sys.argv[1:])
  File "bcbio_nextgen_install.py", line 44, in main
    subprocess.check_call([bcbio, "upgrade"] + _clean_args(sys_argv, args))
  File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/lfaller/local/share/bcbio/anaconda/bin/bcbio_nextgen.py', 'upgrade', '--tooldir=~/local', '--genomes', 'GRCh37', '--aligners', 'bwa', '--aligners', 'bowtie2', '--data']' returned non-zero exit status 1

[chapmanb]

Lina;
I'm also seeing this error intermittently right now, but it comes and goes on retries. I suspect the broad server is busy or having issues this afternoon/evening. Sorry to not have a better suggestion other than waiting and retrying.

[lfaller]

Hi Brad,

I believe you might be correct that the server was just too busy yesterday afternoon. I restarted the install script this morning at 6:30AM EST and the script was able to retrieve the files. So my recommendation would be to only try to install bcbio in the early morning hours... ;-)

Thanks!
~Lina

[lpantano]

Hi,

I will close this since it seems the problem is gone, and only appears intermittently

Thanks!