Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler runs out of memory in nested async/await calls since 1.46 #77628

Closed
sakex opened this issue Oct 6, 2020 · 20 comments
Closed

Compiler runs out of memory in nested async/await calls since 1.46 #77628

sakex opened this issue Oct 6, 2020 · 20 comments
Labels
A-async-await Area: Async & Await C-bug Category: This is a bug. E-needs-mcve Call for participation: This issue needs a Minimal Complete and Verifiable Example I-compilemem Problems and improvements with respect to memory usage during compilation. O-linux Operating system: Linux regression-from-stable-to-stable Performance or correctness regression from one stable version to another.

Comments

@sakex
Copy link

sakex commented Oct 6, 2020

I have a fairly small program (around 5k lines of code), with some heavy dependencies such as Tokio, Actix, Serde and Tokio-Tungstenite. I can't get it to compile outside of Docker because it eats all my RAM + Swap (I have 16Gb.) However, when I compile in Docker with a two steps compilation process, it takes only 1Gb.

My Cargo.toml

serde = {version = "1.0.116", features = ["derive"]}
serde_json = "1.0.58"
reqwest = {version = "0.10.8"}
futures = { version = "0.3.6", default-features = false, features = ["async-await"]}
tokio = {version = "0.2.22", features = ["rt-threaded", "macros", "tcp"]}
itertools = "0.9.0"
dotenv = "0.15.0"
futures-util = { version = "0.3.6", features = ["sink", "std"] }
tokio-tungstenite = {version = "0.10.1", features = ["connect", "tls"]}
tokio-tls = "0.3.1"
hmac = "0.9"
sha2 = "0.9.1"
mongodb = "1.1.1"
bson = "1.1.0"
actix-web = "3.1.0"
actix-rt = "1.1.1"
rustc-serialize = "0.3.24"
chrono = { version = "0.4.19", features = ["serde"] }


[profile.release]
lto = true
codegen-units = 1
opt-level = 3

My Dockerfile:

FROM rust:latest AS build_base

RUN USER=root cargo new --bin myapp
WORKDIR /myapp

COPY Cargo.toml .

RUN cargo build --release
RUN rm -f src/*.rs
COPY ./src ./src
RUN rm ./target/release/deps/myapp*

RUN cargo build --release


FROM rust:latest

WORKDIR /myapp

COPY --from=build_base /myapp/target/release/myapp /myapp/myapp

RUN touch .env

EXPOSE 8000

CMD ["./myapp"]

Meta

rustc --version --verbose:

rustc 1.49.0-nightly (a1dfd2490 2020-10-05)
binary: rustc
commit-hash: a1dfd2490a6cb456b92e469fa550dc217e20ad6d
commit-date: 2020-10-05
host: x86_64-unknown-linux-gnu
release: 1.49.0-nightly
LLVM version: 11.0
@sakex sakex added the C-bug Category: This is a bug. label Oct 6, 2020
@jonas-schievink
Copy link
Member

jonas-schievink commented Oct 6, 2020

What's the Rust version in the Docker container?

@sakex
Copy link
Author

sakex commented Oct 6, 2020

@jonas-schievink It is rust:latest, so I would assume 1.46

@jonas-schievink
Copy link
Member

jonas-schievink commented Oct 6, 2020

Does it still use 16 GB if you use that Rust version outside of the container?

@Aaron1011
Copy link
Member

Aaron1011 commented Oct 7, 2020

@sakex: Is it possible for you to upload your repository? If not, does the high RAM usage occur when building dependencies, or when building your repository?

@sfackler
Copy link
Member

sfackler commented Oct 7, 2020

@jonas-schievink It is rust:latest, so I would assume 1.46

How recently have you docker pulled? Docker does not update images automatically.

@camelid camelid added I-compilemem Problems and improvements with respect to memory usage during compilation. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 7, 2020
@sakex
Copy link
Author

sakex commented Oct 7, 2020

@jonas-schievink, @Aaron1011, @sfackler I actually tried to compile with 1.44.0. It almost didn't use any RAM and it took way less time than I'm used to.

@Aaron1011 I can make give you access to my repo but I'd rather not share it. The code is quite complicated/messy actually.

@sfackler Given my discovery, I will not try to docker pull unless you really need some insights.

Are there any ways I could assist you in finding the regression?

@sakex
Copy link
Author

sakex commented Oct 7, 2020

Update:

I tried with 1.45.0 and 1.46.0. The bug was introduced in 1.46.0.

Is it something you are aware of?

@camelid camelid added the regression-from-stable-to-stable Performance or correctness regression from one stable version to another. label Oct 7, 2020
@rustbot rustbot added the I-prioritize Indicates that prioritization has been requested for this issue label Oct 7, 2020
@camelid camelid added E-needs-mcve Call for participation: This issue needs a Minimal Complete and Verifiable Example and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Oct 7, 2020
@sakex
Copy link
Author

sakex commented Oct 7, 2020

@camelid

Update 2:

I have found the exact line that creates the problem

The original function was:

use bson::{doc};
use bson::oid::ObjectId;
use mongodb::{error::Error, Collection, options::FindOneOptions};

#[derive(Clone)]
pub struct UserService {
    collection: Collection,
}

impl UserService {
    pub async fn get_all(&self) -> Result<Vec<UserSchema>, Error> {
        let mut cursor = self.collection.find(None, None).await?; // That's the bad line
        let mut vec: Vec<UserSchema> = Vec::new();
        while let Some(result) = cursor.next().await {
            match result {
                Ok(document) => {
                    let user: UserSchema = bson::from_bson(bson::Bson::Document(document)).unwrap();
                    vec.push(user);
                }
                Err(e) => return Err(e.into()),
            }
        };
        Ok(vec)
    }
}

By removing chunks of code one by one, I isolated the offending line of code as being:

    let mut cursor = self.collection.find(None, None).await?; // That's the bad line

Which is a call to the MongoDB driver

@camelid camelid added O-linux Operating system: Linux A-async-await Area: Async & Await labels Oct 7, 2020
@sakex
Copy link
Author

sakex commented Oct 7, 2020

Update 3

Lifting the code out of the function inside a .then fixes the issue:


        self.collection.find(None, None)
            .then(|mut res|  async  {
            let mut cursor = res.unwrap();
            let mut users: Vec<UserSchema> = Vec::new();
            while let Some(result) = cursor.next().await {
                match result {
                    Ok(document) => {
                        let trader: UserSchema = bson::from_bson(bson::Bson::Document(document)).unwrap();
                        users.push(trader);
                    }
                    Err(e) => return Err(e),
                }
            };
                Ok(current_traders)
        });

It is of course not so great to do that, but I hope it will help you in finding the bug

@sakex sakex changed the title Compilation uses 16GB of memory on local computer vs 1GB in Docker Compiler runs out of memory in nested async/await calls since 1.46 Oct 7, 2020
@tesuji
Copy link
Contributor

tesuji commented Oct 7, 2020

How long is your normal compile time in 1.44 ?
If possible, since we don't have access to your code,
would you mind running bisect to find a blamed commit ?
The general guide is here: https://github.com/rust-lang/cargo-bisect-rustc/blob/master/TUTORIAL.md.
The start date could be the date starting development of 1.46,
which maybe 2020-06-05.

@lcnr
Copy link
Contributor

lcnr commented Oct 7, 2020

probably related to #75992, currently waiting for an MVCE for that issue

@sakex
Copy link
Author

sakex commented Oct 7, 2020

@lzutao It takes around 1:45 minutes (in debug with dependencies cached)

I will run the bisection now

@tesuji
Copy link
Contributor

tesuji commented Oct 7, 2020

@lzutao It takes around 1:45 minutes (in debug with dependencies cached)

Took much longer than I thought, so bisection is possible, but you might not want
to do it in your local host.

@sakex
Copy link
Author

sakex commented Oct 7, 2020

@lzutao I need to go anyway, so I'll let it run for a few hours, I'll report if I find something

The command:
cargo bisect-rustc --start=2020-07-01 --end=2020-10-07

@LeSeulArtichaut
Copy link
Contributor

LeSeulArtichaut commented Oct 7, 2020

probably related to #75992, currently waiting for an MVCE for that issue

#75992 also regressed in 1.46.0, so this is likely to be a duplicate

@sakex
Copy link
Author

sakex commented Oct 7, 2020

@lzutao


Regression in 0a49057


searched nightlies: from nightly-2020-08-13 to nightly-2020-08-14
regressed nightly: nightly-2020-08-14
searched commits: from 576d27c to 81dc88f
regressed commit: 0a49057

bisected with cargo-bisect-rustc v0.5.2

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc --start=2020-08-13 --end=2020-08-14 

@LeSeulArtichaut
Copy link
Contributor

LeSeulArtichaut commented Oct 7, 2020

This is the same regressed commit as #75992, so closing as a duplicate of #75992.

@LeSeulArtichaut
Copy link
Contributor

LeSeulArtichaut commented Oct 7, 2020

Thank you very much @sakex for taking the time to bisect this!

@camelid
Copy link
Member

camelid commented Oct 8, 2020

Duplicate of #75992.

@camelid
Copy link
Member

camelid commented Oct 8, 2020

Hmm, I thought that would mark it in GitHub...

@camelid camelid removed the I-prioritize Indicates that prioritization has been requested for this issue label Oct 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-async-await Area: Async & Await C-bug Category: This is a bug. E-needs-mcve Call for participation: This issue needs a Minimal Complete and Verifiable Example I-compilemem Problems and improvements with respect to memory usage during compilation. O-linux Operating system: Linux regression-from-stable-to-stable Performance or correctness regression from one stable version to another.
Projects
None yet
Development

No branches or pull requests

9 participants