Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vagrant box downloads extremely slow #5319

Closed
spkane opened this issue Feb 11, 2015 · 112 comments
Closed

Vagrant box downloads extremely slow #5319

spkane opened this issue Feb 11, 2015 · 112 comments

Comments

@spkane
Copy link

@spkane spkane commented Feb 11, 2015

When relying on vagrant to download a box I frequently see connection speeds like this:

default: Downloading: http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
default: Progress: 20% (Rate: 179k/s, Estimated time remaining: 0:41:37)

(Rate: 179k/s)

Yet when I use wget to the same URL:

wget http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
--2015-02-10 09:52:12--  http://boxes.example.com/vagrant/boxes/c6/packer_c6_2.5.2_virtualbox.box
Resolving boxes.example.com... 10.1.0.17
Connecting to boxes.example.com|10.1.0.17|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 830674320 (792M) [text/plain]
Saving to: 'packer_c6_2.5.2_virtualbox.box'

packer_c6_2.5.2_virtualbox.bo   0%[                                                         ]   7.12M   696KB/s   eta 19m 50s

(Rate: 696KB/s) or often higher.

This particular example was pulled when on Wifi and connected to an IPSEC VPN.

@sethvargo

This comment has been minimized.

Copy link
Contributor

@sethvargo sethvargo commented Feb 11, 2015

Hi @spkane

Some boxes are hosted on Atlas and sometimes Atlas is just acting as a proxy to a user-hosted box. If you give more information on the specific box(es) you're downloading, we can do some research.

@spkane

This comment has been minimized.

Copy link
Author

@spkane spkane commented Feb 12, 2015

@sethvargo This box is actually a box I built using packer and it is hosted on a remote server. I'm trying to understand why the download in significantly slower using vagrant then using wget to the exact same URL.

@sethvargo

This comment has been minimized.

Copy link
Contributor

@sethvargo sethvargo commented Feb 12, 2015

@spkane sorry - I misread your original issue.

I would suspect (and maybe @mitchellh could elaborate more) a few things:

  1. Ruby is slow and somehow throttling the subprocess
  2. Wget is faster than curl (which is what Vagrant is using)
  3. Vagrant is also allocating time to unpack the box
  4. Wget is allowing for some type of compressed download

It would be helpful if you could benchmark this with curl for reference.

@mitchellh

This comment has been minimized.

Copy link
Member

@mitchellh mitchellh commented Feb 24, 2015

I really can't explain this. Vagrant doesn't do anything during the subprocess Ruby-wise: it subprocesses to curl. It doesn't even do the download in Ruby. Perhaps wget is using multiple connections to download multiple parts? I really don't know, but unless we get more information I have to assume that Vagrant is fine here.

Is curl just as slow? Vagrant is just subprocessing to curl until it completes.

@mitchellh mitchellh closed this Feb 24, 2015
@xapai

This comment has been minimized.

Copy link

@xapai xapai commented Dec 1, 2015

I'm experience the same slow experience. Anyone can try aria - http://aria2.sourceforge.net/ and http://stackoverflow.com/questions/3430810/wget-download-with-multiple-simultaneous-connections

It's seems a little bit faster, but, man, you can set this up using default vagrant download mechanism and take a walk or make yourself a sandwich. Get way from screen for a little bit.

@karlkfi

This comment has been minimized.

Copy link
Contributor

@karlkfi karlkfi commented Jan 26, 2016

Having the same problem here:

  1. Upload a box manually to atlas
  2. Create a new Vagrantfile with just vm_cfg.vm.box_url = <user>/box-name
  3. vagrant up - box downloads slowly
  4. wget box url from atlas (see vagrant up output) - box downloads lightening fast
@clakeb

This comment has been minimized.

Copy link

@clakeb clakeb commented Feb 1, 2016

I wish there was just a +1 for this. Me too. Same connection for all 3 attempts. VPN turned off.

  • vagrant up took 25+ minutes.
  • wget took 3 minutes.
  • curl took 4 minutes.
@milhaus

This comment has been minimized.

Copy link

@milhaus milhaus commented Feb 2, 2016

Ubuntu vivid64 is downloading at ~56kbps. I'm on a 100mbit symmetric connection.
edit: it timed out before it could finish.
edit2: I can confirm that https://atlas.hashicorp.com/ubuntu/boxes/vivid64/versions/20160128.0.0/providers/virtualbox.box downloads dramatically faster over wget than via "vagrant up".

@ghost

This comment has been minimized.

Copy link

@ghost ghost commented Feb 6, 2016

I'm trying to download the scotch/box and current download speeds using vagrant are less than 10kbps.

default: Progress: 0% (Rate: 2603/s, Estimated time remaining: 33:17:38)

However just as bad using wget.

@tehmaspc

This comment has been minimized.

Copy link
Contributor

@tehmaspc tehmaspc commented Feb 15, 2016

ditto; some popular boxes are very slow to download - i'm updating ubuntu/trusty64 as we speak and it's dropping below 1Kb/s. Been seeing this for a couple wks now.

@JasonTheAdams

This comment has been minimized.

Copy link

@JasonTheAdams JasonTheAdams commented Mar 7, 2016

+1 -- exact same as last comment

@winni2k

This comment has been minimized.

Copy link

@winni2k winni2k commented Mar 9, 2016

Same here:

$ vagrant box add lazygray/heroku-cedar-14
==> box: Loading metadata for box 'lazygray/heroku-cedar-14'
    box: URL: https://atlas.hashicorp.com/lazygray/heroku-cedar-14
==> box: Adding box 'lazygray/heroku-cedar-14' (v1.0.6) for provider: virtualbox
    box: Downloading: https://atlas.hashicorp.com/lazygray/boxes/heroku-cedar-14/versions/1.0.6/providers/virtualbox.box
==> box: Box download is resuming from prior download progress
    box: Progress: 3% (Rate: 281k/s, ...
@k1ng440

This comment has been minimized.

Copy link

@k1ng440 k1ng440 commented Mar 13, 2016

same here

vagrant box update
==> default: Checking for updates to 'laravel/homestead'
    default: Latest installed version: 0.4.1
    default: Version constraints: >= 0
    default: Provider: vmware_desktop
==> default: Updating 'laravel/homestead' with provider 'vmware_desktop' from version
==> default: '0.4.1' to '0.4.2'...
==> default: Loading metadata for box 'https://atlas.hashicorp.com/laravel/homestead'
==> default: Adding box 'laravel/homestead' (v0.4.2) for provider: vmware_desktop
    default: Downloading: https://atlas.hashicorp.com/laravel/boxes/homestead/versions/0.4.2/providers/vmware_desktop.box
    default: Progress: 0% (Rate: 42210/s, Estimated time remaining: 6:10:54))
@richard-scott

This comment has been minimized.

Copy link

@richard-scott richard-scott commented Mar 14, 2016

Is there any way to use something like axel to stream downloads in quicker?

@winni2k

This comment has been minimized.

Copy link

@winni2k winni2k commented Mar 16, 2016

I guess there's nothing preventing people from sharing boxes via torrent. For example, below is a magnet link for the heroku-cedar-14 box:

magnet:?xt=urn:btih:5bb1480d5316f229bb71be55b56b06278de41a67&dn=heroku-cedar-14.box&tr=http%3A%2F%2F9.rarbg.com%3A2710%2Fannounce&tr=http%3A%2F%2Fannounce.torrentsmd.com%3A6969%2Fannounce&tr=http%3A%2F%2Fbt.careland.com.cn%3A6969%2Fannounce&tr=http%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=http%3A%2F%2Fmgtracker.org%3A2710%2Fannounce&tr=http%3A%2F%2Ftracker.tfile.me%2Fannounce&tr=http%3A%2F%2Ftracker.torrenty.org%3A6969%2Fannounce&tr=http%3A%2F%2Ftracker.trackerfix.com%2Fannounce&tr=http%3A%2F%2Fwww.mvgroup.org%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.com%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fglotorrents.pw%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.com%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.glotorrents.com%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.publicbt.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker4.piratux.com%3A6969%2Fannounce

Anyone know a good website where one can search for torrents of vagrant boxes?

@tehmaspc

This comment has been minimized.

Copy link
Contributor

@tehmaspc tehmaspc commented Mar 16, 2016

@wkretzsch - I personally don't know at the moment about any torrent sites - but for me I wouldn't want to trust torrent links as the source for my infrastructure testing. It's a possible option but security is also important. For me official vagrant boxes from folks like puppetlabs hosted on Atlas are so slow to download at times that I wish this issue could be resolved. For internal vagrant boxes that I build for my company we have the option to host on S3 or Artifactory or private Atlas org.

@mitchellh - yes - curl is just as slow (for me). I don't think it is a Vagrant issue - but a backed server hosting issue. Granted - not a Vagrant issue per se.

@faddat

This comment has been minimized.

Copy link

@faddat faddat commented Mar 17, 2016

screenshot from 2016-03-17 14-04-58

Yes, this is because curl can only use one of my 3 connections at the same time. No, that's not the connection's rated speed. The rated speed is 45mbps. Yes, bittorrent does perform better. Just sayin-- your rationale for not supporting bittorrent is kinda thin here.

@winni2k

This comment has been minimized.

Copy link

@winni2k winni2k commented Mar 17, 2016

@tehmaspc surely there must be a way for a website to publish the hash of their box along with a torrent link?

@darkn3rd

This comment has been minimized.

Copy link

@darkn3rd darkn3rd commented Mar 21, 2016

I wish in general, there was a way to have incremental images, like docker images, with vagrant boxes. For the provisioners, which bootstrap (cfengine, chef, salt, puppet, docker, etc) by downloading their platform, I wish there was a way to download a packaged up installer, so that other fresh images that use that provisioner, e.g. ubuntu + docker, would not need to download the goods again. Box updates and provisioner downloads were already painful, but recently, have been beyond notoriously slow.

@PorterBytes

This comment has been minimized.

Copy link

@PorterBytes PorterBytes commented Apr 9, 2016

Just went to update my box for the first time (trusty64 - noticed the warning on my vagrant up command output), and it's going to take my 1.5 hours on a 150MBps connection - pathetic. It's 2016 - I don't know the specifics of what's going on here, but surely we can fix this, like, by the end of next week? The tech that goes into modern technologies like vagrant is amazing, something this basic should be overcome in mere hours.

@faddat

This comment has been minimized.

Copy link

@faddat faddat commented Apr 10, 2016

Amen, Matt, Amen. This is about UX.

There should be a recognition that line speed != line speed and practical
steps can be taken to overcome the daunting issue of line speed != line
speed.

Jacob Gadikian
E-mail: faddat@gmail.com
SKYPE: faddat
Phone/SMS: +84 167 789 6421

On Sun, Apr 10, 2016 at 6:32 AM, Matt Porter notifications@github.com
wrote:

Just went to update my box for the first time (noticed the warning on my
vagrant up command output), and it's going to take my 1.5 hours on a
150MBps connection - pathetic. It's 2016 - I don't know the specifics of
what's going on here, but surely we can fix this, like, by the end of next
week? The tech that goes into modern technologies like vagrant is amazing,
something this basic should be overcome in mere hours.


You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#5319 (comment)

@briancline

This comment has been minimized.

Copy link

@briancline briancline commented Apr 12, 2016

I just tried asking Vagrant to download ubuntu/trusty64, and was getting speeds of <= 5 KiB/sec. I killed it and tried again using the exact same command, and got 29 MiB/sec.

I think @mitchellh is correct in that this doesn't really seem like a Vagrant issue. If anything, it seems more like an Atlas issue (so possibly the ELB and/or whatever's sitting behind it). I highly doubt it has anything to do with the routes or hops between end-users and the ELB VIPs -- you wouldn't typically see such a polarizing set of speeds in that case, especially considering both VIPs terminate in us-east-1.

If for no other reason, it'd be highly desirable to see these made available through a CDN rather than a centrally-located ELB. Then again, I'm just one guy (who isn't paying for this service), so take that for what it's worth. Pretty thankful it's there either way.

@karlkfi

This comment has been minimized.

Copy link
Contributor

@karlkfi karlkfi commented Apr 14, 2016

It's not just an Atlas issue. I have boxes and metadata.json on S3, with a Fastly CDN in front and regularly have the exact same issue: sometimes vagrant downloads at 100kbps and sometimes it downloads at > 5mbps. You can cancel a slow download and half the time a retry gets you the faster speeds.

@milhaus

This comment has been minimized.

Copy link

@milhaus milhaus commented Apr 15, 2016

I contacted support about this around the same time I chimed in here initially. Their response is that Vagrant uses curl to download things so they don't see this as a Vagrant problem. IMO that's an unprofessional cop-out because they chose to use curl, know that there are problems and aren't considering swapping out with an alternative to eliminate the problem for their users.

@Haroenv

This comment has been minimized.

Copy link

@Haroenv Haroenv commented Apr 22, 2016

I can confirm that this is still an issue. All my peers also report times of >1h, while the connection here for other connections is around 200MB/s.

vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'ubuntu/trusty32' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
==> default: Loading metadata for box 'ubuntu/trusty32'
    default: URL: https://atlas.hashicorp.com/ubuntu/trusty32
==> default: Adding box 'ubuntu/trusty32' (v20160406.0.0) for provider: virtualbox
    default: Downloading: https://atlas.hashicorp.com/ubuntu/boxes/trusty32/versions/20160406.0.0/providers/virtualbox.box
    default: Progress: 11% (Rate: 43801/s, Estimated time remaining: 1:36:50)
@faddat

This comment has been minimized.

Copy link

@faddat faddat commented Apr 22, 2016

While I am unsure of the origin of the problem, I really do wish that Hashicorp would get back to its unrelenting focus on user experience with this one. Muli-hour downloads (that should take 1-10 minutes)==bad ux.

@dacodekid

This comment has been minimized.

Copy link

@dacodekid dacodekid commented Apr 25, 2016

Currently downloading an image for the 5th time (@13Xk/s, even with wget). Keep disconnecting me while around 50-90%. But it ALWAYS downloads at full speed either early morning / late night EST. Assuming it is a traffic issue, but regardless very bad UX.

    box: Progress: 47% (Rate: 106k/s, Estimated time remaining: 0:14:50)
@brazitech

This comment has been minimized.

Copy link

@brazitech brazitech commented Apr 29, 2016

I have been trying for 2 day's now and still can not get it to download... its a shame.. it is really not impressing new comers to laravel .. i can only get 34ks speed.........

@atb00ker

This comment has been minimized.

Copy link

@atb00ker atb00ker commented Dec 25, 2017

Are there any trusted torrent links available? I am trying to download bentoo/ubuntu16.04 and i am getting 12 kbps and upto 15kbps when i try to do wget, i am sure that my connection is fine!

@DenzoNL

This comment has been minimized.

Copy link

@DenzoNL DenzoNL commented Dec 27, 2017

Still an issue here, have to update my box at ~150k/s, not able to do a lot of useful work in the meanwhile..

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Dec 29, 2017

Still an issue.

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Dec 29, 2017

@bradisbell The real solution is probably a docker-based development environment. Just waiting for that ecosystem to settle down .

@TimOgilvy

This comment has been minimized.

Copy link

@TimOgilvy TimOgilvy commented Dec 29, 2017

🍺 🍺 🍺 🍺 🍺 🍺

I'm just going to leave these here for any poor sod waiting for this download.

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Dec 29, 2017

I found a root cause and a solution for my situation. I increased my download speed 100x:

Root cause: Slow speed of ubuntu.com cloud box servers
Solution: Switch to another trusted source for Ubuntu boxes, Puppet Labs.

I've been using ubuntu/xenial64 for developing apps that must deploy to Ubuntu 16. (Heroku) And download speeds have become ridiculously slow, or failed entirely. I used wget and watched it follow the redirects and saw that it's actually hitting ubuntu.com. I tried several ways to download from Ubuntu, but they all resulted in speeds around 50 KB/s.

So I went back to the Vagrant box listing to see who else might have a Xenial 64 box, and if I'd trust them. Turns out Puppet does, and I trust them as much as (or even more than) Ubuntu. My Vagrantfile now has the config line:

config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"

And it downloaded in just a few seconds, as opposed to possibly an hour or never.

@tristanmason

This comment has been minimized.

Copy link

@tristanmason tristanmason commented Jan 5, 2018

@dogweather Thanks for this! My download time estimate went from 3 hours to 3 minutes when I switched to a box from the list at https://app.vagrantup.com/puppetlabs/

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Jan 5, 2018

@tristanmason You're welcome!

Ironically: In the past week I switched to Docker Compose, and it's AMAZING. I can't believe I've waited this long to use it. I used this Quickstart for Rails doc.

  • Configuration is 1/10 the size and complexity of Vagrant for my typical Rails app
  • Build and Boot times are much faster
  • On my Linux desktop in particular, there is a 10x speed increase vs. running on Mac.
  • There's a huge reduction in RAM required.
  • I'm going to save a ton of money on development computers. I.e., 8GB RAM is plenty for a Docker-based dev machine. (Not so good for Vagrant+VirtualBox though.) And a top-tier i7 isn't necessary anymore either to get good speeds.

I was able to setup a Docker-based dev environment on four computers in a fraction of the time it takes me to create a single Vagrant setup from scratch.

I'm going to write up a blog with detailed metrics - I'll post it here.

@oncet

This comment has been minimized.

Copy link

@oncet oncet commented Jan 15, 2018

@dogweather Vagrant and Docker seems similar but they are different things. Also, you can use a Docker provider within Vagrant.

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Jan 15, 2018

@oncet Yep, very different! But for my use case, they're equivalent and commodified: Ways I can create and launch an isolated dev environment with just one or two commands.

Docker Compose works great and is far simpler than Vagrant (note, this not just "Docker" per se, which only launches containers one by one) and I'd need to hear about a compelling reason to try a Vagrant+Docker solution, which sounds pretty damn complex. ;-)

@sXe79

This comment has been minimized.

Copy link

@sXe79 sXe79 commented Jan 16, 2018

Pain. This is a pain. I'll never be able to download the 5.0.1 I guess

@jdunk

This comment has been minimized.

Copy link

@jdunk jdunk commented Jan 18, 2018

On a 100mbps connection in Manila, with Vagrant 2.0.1, I was trying to download:

https://vagrantcloud.com/ubuntu/boxes/trusty64/versions/20180110.0.0/providers/virtualbox.box

...and I was getting speeds of anywhere from 1k-150k via vagrant up or even wget. Total download time: 12 hours!

Then I did one thing: I VPN'd to California.

Suddenly, my download took only 3 mins. So that's something worth trying possibly.

@racterub

This comment has been minimized.

Copy link

@racterub racterub commented Jan 26, 2018

Same here, I thought this is a personal problem lol.
Even update box still get really poor download speed, hashicorp please fix :(

==> default: Updating 'ubuntu/xenial64' with provider 'virtualbox' from version
==> default: '20171221.0.0' to '20180122.0.0'...
==> default: Loading metadata for box 'https://vagrantcloud.com/ubuntu/xenial64'
==> default: Adding box 'ubuntu/xenial64' (v20180122.0.0) for provider: virtualbox
    default: Downloading: https://vagrantcloud.com/ubuntu/boxes/xenial64/versions/20180122.0.0/providers/virtualbox.box
==> default: Box download is resuming from prior download progress
    default: Progress: 13% (Rate: 52999/s, Estimated time remaining: 0:31:46)
@dqlopez

This comment has been minimized.

Copy link

@dqlopez dqlopez commented Jan 27, 2018

Ignoring such persistent issue means something on Hashicorp's end, I believe.

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Jan 28, 2018

@dqlopez from the previous comments, this is a solved issue: it's purely the speed of ubuntu's servers. The solution is to use some other org's images.

@fredngo

This comment has been minimized.

Copy link

@fredngo fredngo commented Jan 28, 2018

Confirming that when I used bento/ubuntu-16.04 instead of ubuntu/xenial64 (why?), I got pretty good download speeds. The box downloaded in less than a minute, on a 300 Mbps fibre optic connection. (Unfortunately I didn't think to copy and paste the log.)

@dogweather

This comment has been minimized.

Copy link

@dogweather dogweather commented Feb 4, 2018

I promised I'd report back on my transition to Docker, so here goes. I spent a ton of time creating a config that delivers the one-liner vagrant up experience. It's docker-compose up. And there's only one dependency, Docker. Not two (VM plus Vagrant).

My post about performance of a Docker dev environment on Mac and Linux: medium.com

Repos with my configuration, tailored for Ruby on Rails and Phoenix development. Very very similar. Can be tailored for any language, I'd imagine.

cc: @fredngo

@bradisbell

This comment has been minimized.

Copy link

@bradisbell bradisbell commented Feb 4, 2018

@dogweather Frankly, I don't see how your commentary on docker has anything to do with the issue at hand. Please stop taking this thread off-topic.

@externl

This comment has been minimized.

Copy link

@externl externl commented Feb 4, 2018

Docker is not an alternative to Vagrant. They work completely differently and are intended for different environments. While I'm all for a discussion on the merits of when each should be used, this thread is not really the place for it.

Edit: I love both and use them both in development and production environments.

@jeliasson

This comment has been minimized.

Copy link

@jeliasson jeliasson commented Feb 26, 2018

Haven't read the full thread, but it was a significant difference between using PowerShell and Git Bash.

@mohclips

This comment has been minimized.

Copy link

@mohclips mohclips commented Mar 4, 2018

Well, I cant download a single box tonite. I'm in the UK.

The box 'puppetlabs/ubuntu-16.04-64-puppet' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/puppetlabs/ubuntu-16.04-64-puppet"]
Error: The requested URL returned error: 404 Not Found

config.vm.box = "puppetlabs/ubuntu-16.04-64-nocm"
config.vm.box = "bento/ubuntu-16.04"
config.vm.box = "ubuntu/xenial64"

none of those work.

BUT if you download the box manually via wget, all works fine...

nick@TX200-S5:~/workspaces/elk-vagrant$ wget https://app.vagrantup.com/puppetlabs/boxes/ubuntu-16.04-64-nocm/versions/1.0.0/providers/virtualbox.box
--2018-03-04 00:30:45--  https://app.vagrantup.com/puppetlabs/boxes/ubuntu-16.04-64-nocm/versions/1.0.0/providers/virtualbox.box
Resolving app.vagrantup.com (app.vagrantup.com)... 50.17.237.77, 54.221.226.80, 54.243.175.62, ...
Connecting to app.vagrantup.com (app.vagrantup.com)|50.17.237.77|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://s3.amazonaws.com/puppetlabs-vagrantcloud/ubuntu-16.04-x86_64-virtualbox-nocm-1.0.0.box [following]
--2018-03-04 00:30:46--  https://s3.amazonaws.com/puppetlabs-vagrantcloud/ubuntu-16.04-x86_64-virtualbox-nocm-1.0.0.box
Resolving s3.amazonaws.com (s3.amazonaws.com)... 54.231.32.82
Connecting to s3.amazonaws.com (s3.amazonaws.com)|54.231.32.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 666915336 (636M) [application/vnd.previewsystems.box]
Saving to: ‘virtualbox.box’

100%[==========================================================================================>] 666,915,336 2.23MB/s   in 4m 48s 

2018-03-04 00:35:35 (2.21 MB/s) - ‘virtualbox.box’ saved [666915336/666915336]

nick@TX200-S5:~/workspaces/elk-vagrant$ vagrant box add bento/ubuntu-16.04
The box 'bento/ubuntu-16.04' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/bento/ubuntu-16.04"]
Error: The requested URL returned error: 404 Not Found


nick@TX200-S5:~/workspaces/elk-vagrant$ wget -O bento-ubuntu-16.04 https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
--2018-03-04 00:37:41--  https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
Resolving app.vagrantup.com (app.vagrantup.com)... 54.243.252.123, 50.19.252.69, 54.243.137.45, ...
Connecting to app.vagrantup.com (app.vagrantup.com)|54.243.252.123|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://archivist.vagrantup.com/v1/object/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJib3hlcy80NzYzZDNiMy04Yzk0LTQ2YmMtYTQxNy02MDEwYjkxYzhlZjIiLCJtb2RlIjoiciIsImV4cGlyZSI6MTUyMDEyNDc2MX0.At50HVbqsvj9bfhDrbkzH7G5ON5RCcnYHwm5Xx1GXzA [following]
--2018-03-04 00:37:41--  https://archivist.vagrantup.com/v1/object/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJrZXkiOiJib3hlcy80NzYzZDNiMy04Yzk0LTQ2YmMtYTQxNy02MDEwYjkxYzhlZjIiLCJtb2RlIjoiciIsImV4cGlyZSI6MTUyMDEyNDc2MX0.At50HVbqsvj9bfhDrbkzH7G5ON5RCcnYHwm5Xx1GXzA
Resolving archivist.vagrantup.com (archivist.vagrantup.com)... 50.16.237.173, 23.21.92.233, 54.221.212.171, ...
Connecting to archivist.vagrantup.com (archivist.vagrantup.com)|50.16.237.173|:443... connected.
HTTP request sent, awaiting response... 307 Temporary Redirect
Location: https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/4763d3b3-8c94-46bc-a417-6010b91c8ef2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIA4WZ7ZDX3WM4HDQ%2F20180304%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180304T003742Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=871143b6e5a7078c775fba1f262ad4b3cd31e065ecde0d4ff0c4da2081aec7e7 [following]
--2018-03-04 00:37:42--  https://vagrantcloud-files-production.s3.amazonaws.com/archivist/boxes/4763d3b3-8c94-46bc-a417-6010b91c8ef2?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIA4WZ7ZDX3WM4HDQ%2F20180304%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180304T003742Z&X-Amz-Expires=60&X-Amz-SignedHeaders=host&X-Amz-Signature=871143b6e5a7078c775fba1f262ad4b3cd31e065ecde0d4ff0c4da2081aec7e7
Resolving vagrantcloud-files-production.s3.amazonaws.com (vagrantcloud-files-production.s3.amazonaws.com)... 52.216.164.43
Connecting to vagrantcloud-files-production.s3.amazonaws.com (vagrantcloud-files-production.s3.amazonaws.com)|52.216.164.43|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 428203828 (408M) [binary/octet-stream]
Saving to: ‘bento-ubuntu-16.04’

100%[==========================================================================================>] 428,203,828 2.09MB/s   in 3m 6s  

2018-03-04 00:40:48 (2.19 MB/s) - ‘bento-ubuntu-16.04’ saved [428203828/428203828]

A direct box add works too

nick@TX200-S5:~/workspaces/elk-vagrant$ vagrant box add "bento/ubuntu-16.04" https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
==> box: Box file was not detected as metadata. Adding it directly...
==> box: Adding box 'bento/ubuntu-16.04' (v0) for provider: 
    box: Downloading: https://app.vagrantup.com/bento/boxes/ubuntu-16.04/versions/201802.02.0/providers/virtualbox.box
    box: Progress: 7% (Rate: 2247k/s, Estimated time remaining: 0:03:13)

@mahmoodn

This comment has been minimized.

Copy link

@mahmoodn mahmoodn commented Mar 18, 2018

I would like to know how can I manually add the .box file in an offline manner. It is not clear in the documents on how to do that. I think the correct place for modification is Vagrantfile. However, I can not figure out which parameter should I change.

@externl

This comment has been minimized.

Copy link

@externl externl commented Mar 19, 2018

@mahmoodn Your question is totally unrelated to this thread. The box documentation should help you with your question.

@Silic0nS0ldier

This comment has been minimized.

Copy link

@Silic0nS0ldier Silic0nS0ldier commented Mar 24, 2018

https://vagrantcloud.com/laravel/boxes/homestead/versions/5.2.0/providers/virtualbox.box
    homestead-7: Progress: 36% (Rate: 95492/s, Estimated time remaining: 0:54:16)

99% of the time, this is what I'm dealing with. On a stable fibre connection. No WiFi. Every now and again I get a short burst of speed that pushes progress up around 5%, but overall, I'm left waiting several hours. This is insane. People have proposed work arounds and mitigations but those aren't solutions. When this issue was first closed, it was implied that curl might be to blame. Perhaps this is true, and if it is, then it should be switched out for a different solution.

@YousefSaber

This comment has been minimized.

Copy link

@YousefSaber YousefSaber commented Mar 30, 2018

I suffer the same problem also

My connection on speedtest.net 6.6Mbps while the download takes only 0.6 Mbps of my bandwidth

image

image

@hashicorp hashicorp locked as too heated and limited conversation to collaborators Apr 23, 2018
@kikitux kikitux reopened this Apr 23, 2018
@chrisroberts

This comment has been minimized.

Copy link
Member

@chrisroberts chrisroberts commented Apr 23, 2018

Hi everyone,

This issue overall is very complicated in that it is not easily reproducible and is very dependent on location, network health, and the actual download location of the target box. In many cases the box being downloaded is located on a third party system. One feature that was introduced in Vagrant a couple releases ago was the notification of redirect to show when boxes are being downloaded from a third party server.

There have also been claims that the embedded curl is downloading files slower than the system curl. I have not been successful in validating this claim and do not see a difference in download speed between my system curl and the embedded curl. However, as of Vagrant 2.0.4, the VAGRANT_PREFER_SYSTEM_BINS environment variable is available for all platforms. When enabled, Vagrant will use local system binaries if available instead of the embedded binaries. If you observe faster download speeds with your local curl binary, this provides a switch to easily enable Vagrant to use the local binary.

My apologies for the frustration this issue has caused. It is something I have investigated multiple times, but the number of variables involved makes this extremely difficult to get a valid reproduction. I have updated Vagrant's functionality to make the underlying cause more easily understandable where available (redirect notifications) or easier to work around. I am continuing to investigate our options for box downloads from Vagrant Cloud in different regions of the world (which have occasionally presented with issues). However, with box downloads from third party locations, there is little I am able to do to control their available bandwidth or any kind of throttling they may impose.

Cheers!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.