Cargo not finding deps after upgrade to 1.10

[alexhumphreys]

I'm having trouble using bitbake to compile a project after upgrading to rust 1.10

The error I see is:

DEBUG: Executing python function externalsrc_compile_prefunc
NOTE: sota-client: compiling from external source tree /home/alex/work/pro/poky/meta-oim/recipes-oim/sota-client/../../../../client/rvi_sota_client
DEBUG: Python function externalsrc_compile_prefunc finished
DEBUG: Executing shell function do_compile
NOTE: cargo = /home/alex/work/pro/poky/build/tmp/sysroots/x86_64-linux/usr/bin/cargo
NOTE: rustc = /home/alex/work/pro/poky/build/tmp/sysroots/x86_64-linux/usr/bin/rustc
NOTE: cargo build -v --target x86_64-poky-linux --release
error: no matching package named `getopts` found (required by `sota_client`)
location searched: registry https://github.com/rust-lang/crates.io-index
version required: = 0.2.14
WARNING: exit code 101 from a shell command.
ERROR: Function failed: do_compile (log file is located at /home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/temp/log.do_compile.32405)

Here getopts fails, but every time I try to build it names a different package from the Cargo.toml. If I run it a few more times the packages that fail are: toml, lazy_static, chan, rustc-serialize, etc.

It seems to not have a local cargo registry, or possibly can't find the one that is there?

When I open up a devshell I see $CARGO_HOME=/home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/cargo_home

$ ls $CARGO_HOME
config  git  registry

The config file has a local-registry path that doesn't seem to exist:

cat $CARGO_HOME/config
paths = [
]
[source.local]
local-registry = "/home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/cargo_registry"
[source.crates-io]
replace-with = "local"
registry = "https://github.com/rust-lang/crates.io-index"

$ ls /home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/cargo_registry
ls: cannot access '/home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/cargo_registry': No such file or directory

There's another registry under $CARGO_HOME, but it just has a lockfile:

ls -a $CARGO_HOME/registry/index/github.com-1ecc6299db9ec823/
.  ..  .cargo-index-lock

I get the same result from fresh bitbakes on different machines, and the project builds successfully on 1.7, or 1.10 from multirust.

Any idea what's up? If you need any more info just let me know.

Cheers!

(In case you need it, here's the recipe, and here's the rust project I'm trying to build.)

[codyps]

We've recently modified cargo.bbclass (and cargo itself) to support not doing any fetching itself (all fetching is done in do_fetch). This section of the README has a few details, and for something a bit more concrete, the examples also show the general form for cargo.bbclass recipes to take.

Right now we don't have a mechanism like the old cargo.bbclass (where cargo was allowed to handle fetching itself).

[cardoe]

@alexhumphreys You'll need to use cargo-bitbake to generate your recipe file to get the appropriate dependencies as @jmesmon mentioned above. Let us know when you've tried this or provide a link to an updated recipe so we can help verify this.

[tardyp]

@jmesmon which version of meta-rust would you recommend for use behind proxies?

I would expect that your move of all fetching in do_fetch would resolve the problem, as proxy variables are only exported for those tasks. I think it is a better approach than #83

[cardoe]

You need to use https://crates.io/crates/cargo-bitbake to generate the dependency list for your package. I can see https://github.com/openivimobility/meta-oim/blob/master/recipes-oim/sota-client/sota-client.bb has no dependencies listed out at all which will cause this to fail.

[alexhumphreys]

@jmesmon @cardoe Thanks for the tips, and sorry for not spotting that section of the README.

Also, sorry for commenting on a closed issue, it took a while to get my head around the following.

I've been trying to use the cargo-bitbake command to generate the recipe, but I've been having some trouble.

First (maybe this should be an issue on cargo-bitbake) the script outputs inherit cargo_util, which should probably be inherit cargo

Second, the project I am building is not packaged as a cargo crate, so the generated crate://crates.io/sota_client/0.1.0 won't do much for me. I was previously using externalsrc, so I've had to frankenstein together the output of cargo-bitbake and my rust 1.7 recipe.

I combined them to make the following updated recipe, and I am seeing some strange behaviour.

On a fresh build (bitbake sota-client), this recipe fails with the same error: no matching package named \foo found as above.

If I comment out line 10, #inherit systemd externalsrc so I am only using inherit cargo, and build again, it will fail with ERROR: Function failed: sota-client: LIC_FILES_CHKSUM points to an invalid file: /home/alex/work/pro/poky/build/tmp/work/core2-64-poky-linux/sota-client/1.0-10/sota-client-1.0/README.md, which makes sense; since externalsrc is removed it can't find the README.

However, if I then add back in the inherit systemd externalsrc, so I am back at the updated recipe, and run bitbake sota-client, this build will succeed. So it seems like it needs two runs to build, one to do the cargo stuff, and one to do the externalsrc stuff. By running rm -rf build/tmp/ I've managed to recreate this behaviour at least 3 times.

So yeah, I'm not sure if I'm misconfiguring something again, or if it's that the cargo command doesn't like some stuff being in crates and others being in externalsrc. It would be possible to package the project as a cargo crate and do it that way, but I do like the externalsrc workflow. Also, if this should be a new issue, let me know and I can open one.

Anyway, any pointers you can give me to solve would be greatly appreciated.

[codyps]

@alexhumphreys not being able to use non-crates.io sources combined with crates.io sources definitely sounds like a problem, though I'm not sure how we can solve it when externalsrc is used. externalsrc presumes it can disable fetch/unpack/patch, and with those disabled it's pretty clear that the crate:// fetcher won't run.

Perhaps we need an independent tool to form the cargo repository from crate specs? Or maybe there is a way to provide for some fetching even when externalsrc is used?

An alternate might be to not use externalsrc and instead provide part of SRC_URI that points to the local source, while still keeping all of the crate:// specs.

[cardoe]

@jmesmon I would likely go the second route. There are changes coming to Cargo that will allow us to make a local crates.io repo in the future but that's not here yet and it will require a few improvement iterations. There is a tool to create this repo as well that I believe Mozilla uses and its called https://crates.io/crates/cargo-vendor

@alexhumphreys I see the issue with the inherit. It looks like cargo_util was renamed to cargo proper and I missed that. I'll publish a new version of cargo-bitbake

[alexhumphreys]

@jmesmon Thanks for clearing that up, was wondering why it took two runs. Like @cardoe said, the alternate route where the source is provided via SRC_URI also sounds good to me. Any idea what a vague timeline for that would be (or any other solution) would be?

In the meantime I can either stay on 1.7, or package sota-client as a crate and build it from there.

@cardoe just sat the new cargo-bitbake version, thanks!

[cardoe]

Still working on getting it published into crates.io. Upgrading to macOS Sierra had the fun side effect of breaking my Rust install. So give it a few more minutes hopefully before its published.