cargo not found when building rust-1.15.0 from source

[djc]

I'm trying to update the Gentoo Linux for Rust to 1.15.0. After solving the problems in #39469 with the changes in #39471, I run into a different problem:

python2.7 /var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/bootstrap.py build -v
Traceback (most recent call last):
  File "/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/bootstrap.py", line 492, in <module>
    main()
  File "/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/bootstrap.py", line 475, in main
    rb.build_bootstrap()
  File "/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/bootstrap.py", line 297, in build_bootstrap
    self.run(args, env)
  File "/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/bootstrap.py", line 301, in run
    proc = subprocess.Popen(args, env=env)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
make: *** [Makefile:23: all] Error 1

The command it's trying to execute is ['/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.14.0-x86_64-unknown-linux-gnu/rustc/bin/cargo', 'build', '--manifest-path', '/var/tmp/portage/dev-lang/rust-1.15.0/work/rustc-1.15.0-src/src/bootstrap/Cargo.toml', '--frozen'].

That is, it tries to find cargo in the downloaded rustc-1.14.0-x86_64-unknown-linux-gnu, but doesn't find it there. How is this supposed to work with the new build system? Is this stuff documented anywhere?

[alexcrichton]

It looks like you're running the build system against a directly downloaded release tarball, is that right? If so then it's not the same structure as a standard installation, so you'll have to manually configure where to find cargo (in the same way you're manually configuring where to find rustc presumably)

[icorderi]

@djc, I just run into this same problem too (we are using buildroot).

If you take a look at the bootstrap.py it doesn't need us to use an older version of rust to bootstrap anymore. The new script is capable of downloading its own stage0.

Your build script is probably hooking things up with something similar to:

# old setup, uses an external bootstrap package
define HOST_RUST_BUILDROOT_CONFIGURE_CMDS
	(cd $(@D); $(HOST_CONFIGURE_OPTS) \
		$(HOST_RUST_BUILDROOT_CONF_ENV) \
		./configure \
		--target=$(GNU_TARGET_NAME) \
		--prefix="$(HOST_DIR)/usr" \
		--jemalloc-root="$(HOST_DIR)/usr/lib" \
		--enable-local-rust \
		--local-rust-root="$(HOST_RUST_BOOTSTRAP_DIR)/rustc" \
		--disable-docs \
		--disable-manage-submodules \
		--sysconfdir="$(HOST_DIR)/etc" \
		--localstatedir="$(HOST_DIR)/var/lib" \
		--datadir="$(HOST_DIR)/usr/share" \
		--infodir="$(HOST_DIR)/usr/share/info" \
		$(HOST_RUST_BUILDROOT_CONF_OPTS) \
	)
endef

You can get rid of the following two lines, along with the rust-bootstrap package.

                --enable-local-rust \
		--local-rust-root="$(HOST_RUST_BOOTSTRAP_DIR)/rustc" \

fyi: we are now trying to get past errors compiling alloc_jemalloc, there might be some other tweaks needed

cc: @elebihan, seems rust-boostrap and host-cargo-bootstrap packages can be removed from buildroot packages.

[icorderi]

Well, seems the jemalloc issue was known (#35349) and patched (#39392) but apparently it was not cherry-picked to 1.15.

That jemalloc-root config line needs to point to a file not a directory:

--jemalloc-root="$(HOST_DIR)/usr/lib/libjemalloc_pic.a" \

The next set of errors are on the stage1 for rustc-main, it's gonna be a long day...

[icorderi]

Took us quite a few days to get the compilation from source back to work with cross compilation support.

Configuration

define HOST_RUST_BUILDROOT_CONFIGURE_CMDS
	(cd $(@D); $(HOST_CONFIGURE_OPTS) \
		$(HOST_RUST_BUILDROOT_CONF_ENV) \
		./configure \
		--target=$(GNU_TARGET_NAME) \
		--prefix="$(HOST_DIR)/usr" \
		--disable-docs \
		--disable-manage-submodules \
		--sysconfdir="$(HOST_DIR)/etc" \
		--localstatedir="$(HOST_DIR)/var/lib" \
		--datadir="$(HOST_DIR)/usr/share" \
		--infodir="$(HOST_DIR)/usr/share/info" \
		$(HOST_RUST_BUILDROOT_CONF_OPTS) \
	)
        # The following is to get mips64 working
	(cd $(@D); \
		echo > config.toml; \
		echo "[rust]" >> config.toml; \
		echo "backtrace = false" >> config.toml; \
		echo "[target.$(GNU_TARGET_NAME)]" >> config.toml; \
		echo "cc = \"$(GNU_TARGET_NAME)-gcc\"" >> config.toml; \
		echo "cxx = \"$(GNU_TARGET_NAME)-g++\"" >> config.toml; \
	)
endef

Build step

@alexcrichton, the make did't allow us to pass arguments down to the script. I'm not clear what's the intended way to get rust built for the host and target

We had to call the bootstrap.py script directly:

define HOST_RUST_BUILDROOT_BUILD_CMDS
	(cd $(@D); \
	 $(HOST_MAKE_ENV) $(HOST_RUST_BUILDROOT_MAKE_ENV) $(HOST_DIR)/usr/bin/python \
		src/bootstrap/bootstrap.py build $(if $(VERBOSE),-v); \
	 $(HOST_MAKE_ENV) $(HOST_RUST_BUILDROOT_MAKE_ENV) $(HOST_DIR)/usr/bin/python \
		src/bootstrap/bootstrap.py build --keep-stage 1 --target $(GNU_TARGET_NAME) $(if $(VERBOSE),-v); \
	)
endef

[alexcrichton]

Yes the current Makefile is intended to just be a thin veneer of x.py to be familiar to those who desire a ./configure && make workflow. Fore more advanced configuration you'll need to dip beyond directly into x.py arguments or into config.toml configuration

[icorderi]

Got it, thanks!

[elebihan]

@icorderi I'll keep the rust-bootstrap and cargo-bootstrap packages for Buildroot, because they are needed for offline builds. I've updated the rust package to use rustbuild. The installation step is a bit tricky because rustbuild in 1.15.1 can't handle the "prefix" section of config.toml and the target libstd is not installed when using ./x.py dist --install (see #39235).

[Mark-Simulacrum]

Is there something left to solve here? I'm not seeing anything specific...

[Mark-Simulacrum]

I'm going to go ahead and close, please let us know if there's still something pending here.