The server takes up a lot of memory after hundreds of millions of requests

[zzl221000]

Bug Report

Version

axum v0.2.4

Platform

docker image base distroless cc-debian10

Crates

axum="0.2.4"
hyper = { version = "0.14.11", features = ["full"] }
tokio = { version = "1.10.0", features = ["full"] }
tower = { version = "0.4", features = ["full"] }
tower-http = { version = "0.1", features = ["full" ] }
tracing = "0.1"
tracing-subscriber = "0.2"

Description

Using ConnectInfo to get remote ip takes up a lot of memory after hundreds of millions of requests.

use axum::handler::get;
use std::{
    net::SocketAddr,
};
use axum::{ Router};
use tower_http::{
    trace::TraceLayer,
};
use axum::extract::ConnectInfo;

#[tokio::main]
async fn main() {
    if std::env::var("RUST_LOG").is_err() {
        std::env::set_var("RUST_LOG", "INFO,tower_http=DEBUG")
    }
    tracing_subscriber::fmt::init();

    let app = Router::new()
        .route("/ip", get(get_ip))

        // Add middleware to all routes

        .layer(TraceLayer::new_for_http())
        ;

    let addr = SocketAddr::from(([0, 0, 0, 0], 8380));
    tracing::info!("listening on {}", addr);
    axum::Server::bind(&addr)
        .serve(app.into_make_service_with_connect_info::<SocketAddr, _>())
        .await
        .unwrap();
}


async fn get_ip(
    ConnectInfo(addr): ConnectInfo<SocketAddr>
) -> String {
    format!("{}", addr.ip())
}

[davidpdrsn]

Can you provide more context? How many requests? How much memory? Did removing ConnectInfo change things? Can you reproduce it using hyper directly without axum?

I have tested the code you posted and saw no increase in memory with increasing the number of requests.

My numbers are

• 10.000 requests: 5,8 mb
• 100.000 requests: 7,0 mb
• 1.000.000 requests: 6,3 mb
• 10.000.000 requests: 7,0 mb
• 100.000.000 requests: 7,7 mb

The code sending the requests was:

use futures::prelude::*;
use hyper::{Body, Client, Request};
use std::sync::atomic::{AtomicU64, Ordering};

static COUNT: AtomicU64 = AtomicU64::new(0);
const N: u64 = 100_000_000;

#[tokio::main]
async fn main() {
    println!("N = {}", N);

    std::thread::spawn(|| {
        while COUNT.fetch_add(1, Ordering::Relaxed) < N {
            println!(
                "{}",
                ((COUNT.load(Ordering::Relaxed) as f64 / N as f64) * 100.0).round() as i64
            );
            std::thread::sleep_ms(1000);
        }
    });

    let tasks = (0..100)
        .map(|_| {
            tokio::spawn(async move {
                let client = Client::new();
                loop {
                    if COUNT.fetch_add(1, Ordering::Relaxed) >= N {
                        break;
                    }

                    client
                        .request(
                            Request::builder()
                                .uri("http://localhost:8380/ip")
                                .body(Body::empty())
                                .unwrap(),
                        )
                        .await
                        .unwrap();
                }
            })
        })
        .collect::<Vec<_>>();

    for t in tasks {
        t.await.unwrap();
    }
}

The server code is what you posted.

[zzl221000]

@davidpdrsn
I use a proxy to access the service. There are one million proxies every day. They are different remote ips and do not repeat every day.

How many requests?

2,000 * 60 * 60 * 24=172,800,000 requests/day.

How much memory?

The service used 6MB of memory when it was started, 500MB of memory was used after a day of requests, and 2GB of memory was used in about three days.

Did removing ConnectInfo change things?

No, I need to get remote ip. Maybe you can provide a way to get remote ip without using ConnectInfo, I would test it.

Can you reproduce it using hyper directly without axum?

Ok, the result will be given to you in 48 hours.

[davidpdrsn]

No, I need to get remote ip. Maybe you can provide a way to get remote ip without using ConnectInfo, I would test it.

Thats not possible due to hyper's design.

Ok, the result will be given to you in 48 hours.

Can you share code for that as well?

In general the code you posted is doing very little so if there is an issue I it is unlikely axum is the cause.

[zzl221000]

@davidpdrsn

Can you share code for that as well?

use std::net::SocketAddr;
use std::convert::Infallible;
use hyper::{Body, Response, Server};
use hyper::service::{make_service_fn, service_fn};
use hyper::server::conn::AddrStream;

async fn show_headers(addr: SocketAddr) -> Result<Response<Body>, Infallible> {
    Ok(Response::new( format!("{}",addr.ip()).into()))
}

#[tokio::main]
async fn main() {
    let make_service =
        make_service_fn(move |conn: &AddrStream| {
            let addr = conn.remote_addr();
            async move {
                let addr = addr.clone();
                Ok::<_, Infallible>(service_fn(move |_| show_headers(addr.clone())))
            }
        });
    Server::bind(&SocketAddr::from(([0, 0, 0, 0], 8380)))
        .serve(make_service)
        .await
        .unwrap();
}

Cargo.toml

[dependencies]
hyper = { version = "0.14", features = ["full"] }
tokio = { version = "1", features = ["full"] }

Result

hyper version is ok. axum used too much memory.

root     2337225  0.8  0.0 547076 11364 ?        Ssl  09:42   4:14 /hyper-ip-find
root     2395681  1.1  0.6 557496 96048 ?        Ssl  13:25   3:01 /ip-find

[davidpdrsn]

Are you able to make a reproduction script that I can run?

[zzl221000]

Are you able to make a reproduction script that I can run?

python script ,need python >= 3.8
The following script can reproduce this problem. You cannot run it because the proxy IP cannot be shared.
@davidpdrsn
requirements.txt:

• aiohttp

import asyncio
import time
from asyncio import TimeoutError
from attr import attrs, attr
import aiohttp
from aiohttp import ClientProxyConnectionError, ServerDisconnectedError, ClientOSError, ClientHttpProxyError

EXCEPTIONS = (
    ClientProxyConnectionError,
    ConnectionRefusedError,
    TimeoutError,
    ServerDisconnectedError,
    ClientOSError,
    ClientHttpProxyError,
    AssertionError
)
TEST_URL = 'http://localhost:8380/ip'
TEST_VALID_STATUS = [200]
TEST_TIMEOUT = 10


@attrs(hash=True)
class Proxy(object):
    """
    proxy schema
    """
    host = attr(type=str, default=None, hash=True)
    port = attr(type=int, default=None, hash=True)

    @staticmethod
    def of(raw_ip: str):
        ris = raw_ip.split(':')
        return Proxy(ris[0], int(ris[1]))

    def __str__(self):
        """
        to string, for print
        :return:
        """
        return f'{self.host}:{self.port}'

    def string(self):
        """
        to string
        :return: <host>:<port>
        """
        return self.__str__()


async def check_batch(ips: list[Proxy]):
    results = await asyncio.gather(*[check_single(ip) for ip in ips])
    return [r for r in results if r]


async def check_single(proxy: Proxy):
    try:
        reader, writer = await asyncio.wait_for(asyncio.open_connection(
            proxy.host, proxy.port), timeout=5)
        writer.close()
    except Exception:
        return proxy

    async with aiohttp.ClientSession(connector=aiohttp.TCPConnector(ssl=False)) as session:
        try:
            async with session.get(TEST_URL, proxy=f'http://{proxy.string()}', timeout=TEST_TIMEOUT,
                                   allow_redirects=False) as response:
                if response.status in TEST_VALID_STATUS:
                    return
                else:
                    return proxy
        except EXCEPTIONS:
            return proxy
        except Exception as e:
            return proxy


def load_proxy():
    # load all proxy
    return []


if __name__ == '__main__':
    while True:
        proxy_list = load_proxy()
        asyncio.run(check_batch(proxy_list))
        time.sleep(1)

[davidpdrsn]

The following script can reproduce this problem. You cannot run it because the proxy IP cannot be shared.

I don't understand. Can I run the script or not? It's hard for us to figure out what's wrong if the bug cannot be reproduced.

[zzl221000]

def load_proxy():
    # load all http proxy  from you datasource
    return []

http proxies are required to run this script and reproduce this issue.
My http proxy is a paid service, I can’t provide it.
@davidpdrsn

[davidpdrsn]

Do you think using an HTTP proxy actually matters? If the problem is caused by axum I suppose it shouldn't matter.

[zzl221000]

@davidpdrsn Very little memory usage when not using http proxy.

[davidpdrsn]

Alright I guess thats good. How do you suggest we debug the issue then?

[zzl221000]

@davidpdrsn Maybe I can use jemalloc to dump the memory and submit it. Or is there a better way to debug rust memory?

[jplatte]

You could try heaptrack or massif.

[zzl221000]

@davidpdrsn
This is heaptrack result

[davidpdrsn]

@zzl221000 Do you see anything in that from axum? I've never used it before.

[zzl221000]

Do you see anything in that from axum? I've never used it before.
@davidpdrsn After configuring the environment of the heaptrack gui, I will share the result.

[davidpdrsn]

@zzl221000 any news?

[zzl221000]

@davidpdrsn I can't continue working until the day after I'm on vacation.

I found the same issue in the hyper project. But that's a issue that has been solved.

[zzl221000]

@davidpdrsn It‘s hyper‘s bug.hyper version program had same problem after running for seven days.

root     2337225  1.3  0.1 549100 30020 ?        Ssl  Sep27 200:59 /hyper-ip-check

Memory leak on high number of concurrent connections

[davidpdrsn]

Alright good to know! I'll close this for now but suggest you re-open that hyper issue or file a new one.