Compiler runs out of memory in nested async/await calls since 1.46

[sakex]

I have a fairly small program (around 5k lines of code), with some heavy dependencies such as Tokio, Actix, Serde and Tokio-Tungstenite. I can't get it to compile outside of Docker because it eats all my RAM + Swap (I have 16Gb.) However, when I compile in Docker with a two steps compilation process, it takes only 1Gb.

My Cargo.toml

serde = {version = "1.0.116", features = ["derive"]}
serde_json = "1.0.58"
reqwest = {version = "0.10.8"}
futures = { version = "0.3.6", default-features = false, features = ["async-await"]}
tokio = {version = "0.2.22", features = ["rt-threaded", "macros", "tcp"]}
itertools = "0.9.0"
dotenv = "0.15.0"
futures-util = { version = "0.3.6", features = ["sink", "std"] }
tokio-tungstenite = {version = "0.10.1", features = ["connect", "tls"]}
tokio-tls = "0.3.1"
hmac = "0.9"
sha2 = "0.9.1"
mongodb = "1.1.1"
bson = "1.1.0"
actix-web = "3.1.0"
actix-rt = "1.1.1"
rustc-serialize = "0.3.24"
chrono = { version = "0.4.19", features = ["serde"] }


[profile.release]
lto = true
codegen-units = 1
opt-level = 3

My Dockerfile:

FROM rust:latest AS build_base

RUN USER=root cargo new --bin myapp
WORKDIR /myapp

COPY Cargo.toml .

RUN cargo build --release
RUN rm -f src/*.rs
COPY ./src ./src
RUN rm ./target/release/deps/myapp*

RUN cargo build --release


FROM rust:latest

WORKDIR /myapp

COPY --from=build_base /myapp/target/release/myapp /myapp/myapp

RUN touch .env

EXPOSE 8000

CMD ["./myapp"]

Meta

rustc --version --verbose:

rustc 1.49.0-nightly (a1dfd2490 2020-10-05)
binary: rustc
commit-hash: a1dfd2490a6cb456b92e469fa550dc217e20ad6d
commit-date: 2020-10-05
host: x86_64-unknown-linux-gnu
release: 1.49.0-nightly
LLVM version: 11.0

[jonas-schievink]

What's the Rust version in the Docker container?

[sakex]

@jonas-schievink It is rust:latest, so I would assume 1.46

[jonas-schievink]

Does it still use 16 GB if you use that Rust version outside of the container?

[Aaron1011]

@sakex: Is it possible for you to upload your repository? If not, does the high RAM usage occur when building dependencies, or when building your repository?

[sfackler]

@jonas-schievink It is rust:latest, so I would assume 1.46

How recently have you docker pulled? Docker does not update images automatically.

[sakex]

@jonas-schievink, @Aaron1011, @sfackler I actually tried to compile with 1.44.0. It almost didn't use any RAM and it took way less time than I'm used to.

@Aaron1011 I can make give you access to my repo but I'd rather not share it. The code is quite complicated/messy actually.

@sfackler Given my discovery, I will not try to docker pull unless you really need some insights.

Are there any ways I could assist you in finding the regression?

[sakex]

Update:

I tried with 1.45.0 and 1.46.0. The bug was introduced in 1.46.0.

Is it something you are aware of?

[sakex]

@camelid

Update 2:

I have found the exact line that creates the problem

The original function was:

use bson::{doc};
use bson::oid::ObjectId;
use mongodb::{error::Error, Collection, options::FindOneOptions};

#[derive(Clone)]
pub struct UserService {
    collection: Collection,
}

impl UserService {
    pub async fn get_all(&self) -> Result<Vec<UserSchema>, Error> {
        let mut cursor = self.collection.find(None, None).await?; // That's the bad line
        let mut vec: Vec<UserSchema> = Vec::new();
        while let Some(result) = cursor.next().await {
            match result {
                Ok(document) => {
                    let user: UserSchema = bson::from_bson(bson::Bson::Document(document)).unwrap();
                    vec.push(user);
                }
                Err(e) => return Err(e.into()),
            }
        };
        Ok(vec)
    }
}

By removing chunks of code one by one, I isolated the offending line of code as being:

    let mut cursor = self.collection.find(None, None).await?; // That's the bad line

Which is a call to the MongoDB driver