New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to mmap #2808

Closed
miedzinski opened this Issue Jun 27, 2016 · 30 comments

Comments

Projects
None yet
10 participants
@miedzinski

miedzinski commented Jun 27, 2016

I can't install anything with cargo.

$ cargo install --verbose rustfmt
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2/-1] Failed to mmap. Could not write data: Invalid argument

or by putting dependency in Cargo.toml

$ cat Cargo.toml
[package]
name = "test"
version = "0.1.0"
authors = ["test"]

[dependencies]
rustfmt = "0.5"
$ cargo build --verbose
    Updating registry `https://github.com/rust-lang/crates.io-index`
error: Unable to update registry https://github.com/rust-lang/crates.io-index

Caused by:
  failed to fetch `https://github.com/rust-lang/crates.io-index`

Caused by:
  [2/-1] Failed to mmap. Could not write data: Invalid argument
$ rustc --version
rustc 1.9.0 (e4e8b6668 2016-05-18)
$ cargo --version
cargo 0.10.0-nightly (10ddd7d 2016-04-08)

I am running up-to-date Ubuntu 16.04 LTS.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 27, 2016

Whoa, haven't seen that one before! Are you perhaps running on a system that's running low on memory? It may be worth running strace as well to take a look at the parameters to the mmap call that's failing.

@miedzinski

This comment has been minimized.

miedzinski commented Jun 27, 2016

mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)

There is also one thing I forgot to mention and it may be the cause. That Ubuntu is inside VirtualBox 5.0.22 on Windows host. I'm using this setup for years and never had any problems, so I haven't thought about mentioning this in first place.

Definitely RAM usage isn't an issue (host or guest).


This may be related to #1578. My whole home dir is on vbox shared folder.

$ df -T
Filesystem     Type     1K-blocks      Used Available Use% Mounted on
udev           devtmpfs    490492         0    490492   0% /dev
tmpfs          tmpfs       101592      3272     98320   4% /run
/dev/sda1      ext4      32894832   8020564  23180268  26% /
tmpfs          tmpfs       507956         8    507948   1% /dev/shm
tmpfs          tmpfs         5120         0      5120   0% /run/lock
tmpfs          tmpfs       507956         0    507956   0% /sys/fs/cgroup
/home          vboxsf   524287996 220802040 303485956  43% /home/dominik
tmpfs          tmpfs       101592         0    101592   0% /run/user/1000

I'm not sure whether it's important, but i'm using git from guest without any issues.


libgit2/git2go#194 (comment) looks familiar.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 27, 2016

Interesting! Could you gist the full syscall log as well? It looks like it's attempting to map whatever file descriptor 5 is, and it may be good to see how that's in play

@miedzinski

This comment has been minimized.

miedzinski commented Jun 27, 2016

https://paste.ubuntu.com/17997204/

I guess it's /home/dominik/.cargo/registry/index/github.com-88ac128001ac3a9a/.git/objects/pack/pack_git2_iAED2h/

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 27, 2016

cc @carlosmn, does this sound familiar in libgit2?

It looks like the relevant syscalls leading up to the failure here are:

open("/home/dominik/.cargo/registry/index/github.com-88ac128001ac3a9a/.git/objects/pack/pack_git2_iAED2h", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[..]
lseek(5, 10, SEEK_SET)                  = 10
write(5, "P", 1)                        = 1
mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)

I think that cargo is using libgit2 0.2.23 right now FWIW

@carlosmn

This comment has been minimized.

carlosmn commented Jun 28, 2016

It looks to be the same issue as the one in the git2go tracker. The VirtualBox filesystem does not like something we're doing, but it's not clear what.

It's possible that it doesn't support files with holes even though it doesn't actually complain about them, or maybe it's just with mmap. Someone reported the tests run fine, which is really odd as it's the same code. It probably doesn't help that it seems to be translating unix calls into Windows filesystem calls, which breaks the assumptions we're making in the library.

@miedzinski can you clone fine if you're on VBox but without a shared home dir?

@miedzinski

This comment has been minimized.

miedzinski commented Jun 28, 2016

@carlosmn I can use all git commands (including clone) on vboxsf. If you are asking about libgit2 (which git binary doesn't use, right?), then you have to wait few hours until I come back home and try it with cargo.

@miedzinski

This comment has been minimized.

miedzinski commented Jun 28, 2016

@carlosmn, @alexcrichton, it works flawlessly.

$ cargo install rustfmt
    Updating registry `https://github.com/rust-lang/crates.io-index`
 Downloading rustfmt v0.5.0
# downloading and compiling lots of crates
  Installing /home/libgit/.cargo/bin/rustfmt
  Installing /home/libgit/.cargo/bin/cargo-fmt
$ ~/.cargo/bin/rustfmt -V
0.5.0

This is the same virtual machine, but this time cargo directory is on ext4.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 28, 2016

@miedzinski could you try compiling this and see if it runs into the same error when cloning onto a shared directory?

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "https://github.com/rust-lang/crates.io-index",
                   "crates.io-index",
                   NULL) == 0);
}
@miedzinski

This comment has been minimized.

miedzinski commented Jun 28, 2016

a.out: test.c:10: main: Assertion git_clone(&repo, "https://github.com/rust-lang/crates.io-index", "crates.io-index", NULL) == 0' failed.`

Built against 0.24.1-2 from Ubuntu repository.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 28, 2016

@miedzinski could you try using strace as well to ensure that it's failing with the same mmap error as before?

@miedzinski

This comment has been minimized.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 29, 2016

Hm it's not clear from that gist that it's the same mmap error you're hitting with Cargo. It's also doing a different operation with a clone rather than a fetch, so that may be factoring in as well. It may be worth poking around various incantations to see if it can be reproduced with just libgit2 so we can report the bug upstream.

@carlosmn

This comment has been minimized.

carlosmn commented Jun 29, 2016

That error looks like libgit2 was not built with HTTPS support. Trying with git:// might provide more reliable results.

I've opened libgit2/libgit2#3845 to track this upstream, but I'm not sure that there's a fix that doesn't involve a lot of yak shaving so we don't need to read the data back from disk.

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 29, 2016

Sounds good to me, thanks @carlosmn!

@miedzinski

This comment has been minimized.

miedzinski commented Jun 29, 2016

@carlosmn, there is definitely something wrong here. I have changed this code to:

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "git://git@github.com:rust-lang/crates.io-index.git"
                   "crates.io-index",
                   NULL) == 0);
}

and assertion was failing as well. I have tried different URLs (like git@github.com:rust-lang/crates.io-index.git) and after few tries I saw crates.io-index directory (it could have been there since first execution though). And what's weird, I couldn't delete it -- rm was complaining that it can't remove pack_xxx file with error Operation not permitted. I have deleted it straight from Windows, but since then I couldn't make libgit2 to create this directory again.
Like I said few times, official git binary produces valid repositories.

@alexcrichton, is it possible to change cargo directory location? It seems like a good workaround. If not, I can always mount ext4 filesystem on ~/.cargo/.

edit: symlink is sufficient. :)

@carlosmn

This comment has been minimized.

carlosmn commented Jun 29, 2016

"git://git@github.com:rust-lang/crates.io-index.git"

That's not going to work, since you're aksing for rust-lang to be the port. You're mixing URLs with scp/rsync syntax.

"Operation not permitted" is another error that seems to come from weird interactions between vboxfs and Windows. I'm not sure where the error comes from in that case.

@miedzinski

This comment has been minimized.

miedzinski commented Jun 29, 2016

It seems you are right. I have found which URL produced these files.

#include <git2.h>
#include <assert.h>

int main() {
  git_repository *repo;
  assert(git_libgit2_init() == 1);
  assert(git_clone(&repo,
                   "git://github.com/rust-lang/crates.io-index",
                   "crates.io-index",
                   NULL) == 0);
}

Unlike https version, this creates crates.io-index directory:

$ tree -a crates.io-index/
crates.io-index/
└── .git
    ├── objects
    │   └── pack
    │       └── pack_git2_Hfr6Me
    └── refs
        ├── heads
        └── tags

6 directories, 1 file

But assertion still fails. This time strace shows that unlink call returns Operation not permitted. I can't remove it from inside virtual machine as well.

# rm -rf crates.io-index/
rm: cannot remove 'crates.io-index/.git/objects/pack/pack_git2_Hfr6Me': Operation not permitted

strace ouput: https://paste.ubuntu.com/18115479/

@alexcrichton

This comment has been minimized.

Member

alexcrichton commented Jun 29, 2016

Oh that looks like it's a reproduction, from the strace logs:


open("/home/dominik/Projects/crates.io-index/.git/objects/pack/pack_git2_7Ak1Xv", O_RDWR|O_CREAT|O_EXCL, 0600) = 5
[..]
lseek(5, 10, SEEK_SET)                  = 10
write(5, "P", 1)                        = 1
mmap(NULL, 11, PROT_READ|PROT_WRITE, MAP_SHARED, 5, 0) = -1 EINVAL (Invalid argument)
@miedzinski

This comment has been minimized.

miedzinski commented Jun 29, 2016

Well, there is problem with chmods on vboxsf, because Windows doesn't support them. Using chmod (root or normal user) has no effect. My entry in /etc/fstab looks like this:

home /home/dominik vboxsf defaults,uid=dominik,gid=dominik,fmask=077,dmask=077 0 0

Which means all files have chmod 700 inside my $HOME directory.

@lilith

This comment has been minimized.

Contributor

lilith commented Aug 19, 2016

With Rust 1.11 & cargo 1.12, I get what appears to be a related error:

    Updating registry`https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
warning: spurious network error (1 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
error: failed to update registry https://github.com/rust-lang/crates.io-index

@splix

This comment has been minimized.

splix commented Feb 5, 2017

I have same issue with cargo 1.16 and rust 1.15. The only difference that I'm using KVM+QEMU virtual machine, but with shared home folder as well. My host machine is on Debian, as well as guest.
If I change home folder to local, instead of shared, all works fine.

@Nercury

This comment has been minimized.

Nercury commented Feb 14, 2017

I think I have hit the same issue on OSX while wanting to be clever and reuse cargo registry between builds on a docker:

docker run \
    -v $(pwd)/build/.cargo:/root/.cargo \
    -v $(pwd):/source jimmycuadra/rust:1.15.0 \
    /bin/bash -c "cargo build --release"
    Updating registry `https://github.com/rust-lang/crates.io-index`
warning: spurious network error (2 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
warning: spurious network error (1 tries remaining): [2/-1] Failed to mmap. Could not write data: Invalid argument
error: failed to fetch `https://github.com/rust-lang/crates.io-index`
@miedzinski

This comment has been minimized.

miedzinski commented Feb 14, 2017

This issue is mostly related to vboxsf filesystem. Consider running cargo with strace (look at @alexcrichton comments above) and report to libgit2/libgit2#3845 if it fails the same way on overlayfs (or whatever docker is using right now) or create new issue.

@jml

This comment has been minimized.

jml commented Mar 3, 2017

@Nercury Did you come up with a clever way of re-using registry without triggering this bug?

@Nercury

This comment has been minimized.

Nercury commented Mar 3, 2017

@jml Sorry, no. My project is very small so this is minor for me.

@shepmaster

This comment has been minimized.

Member

shepmaster commented Jul 20, 2017

Following the threads further, @jml's solution was to use a partial cache of the relevant pieces:

# We want to save the whole of /cargo between runs, but if we do that by
# sharing a mountpoint with the local filesystem, we hit a bug in libgit2 vs
# vboxsf:
#
# - https://github.com/rust-lang/cargo/issues/2808
# - https://github.com/libgit2/libgit2/issues/3845
#
# Instead, share the registry/cache, which is where most of the data is.

docker run ... -v $(REGISTRY_CACHE):/cargo/registry/cache ...

@omarabid

This comment has been minimized.

omarabid commented Jul 20, 2017

@shepmaster I'm not getting any speed improvements using it. The cache folder is also empty. Mind that I only have one dependency which is "Rand" so I wonder if the cache is used for other stuff?

@jml

This comment has been minimized.

jml commented Jul 21, 2017

Sorry haven't had a chance to look into this in a while and don't expect to any time soon. I've lost all mental context.

@carols10cents

This comment has been minimized.

Member

carols10cents commented Sep 25, 2017

It sounds like there are either workarounds or upstream issues for this, so I'm going to close. Please reopen if I'm incorrect and there's something cargo should be doing here!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment