From 6ce830706054700db778134e7daa4426d582891c Mon Sep 17 00:00:00 2001 From: Rob Ede Date: Thu, 24 Aug 2023 00:56:31 +0100 Subject: [PATCH] remove num_cpus dependency (#488) --- .github/workflows/ci.yml | 4 ++ actix-server/CHANGES.md | 88 ++++++++++++++++++------------------- actix-server/Cargo.toml | 1 - actix-server/src/builder.rs | 10 ++++- actix-server/src/worker.rs | 8 +++- scripts/free-disk-space.sh | 53 ++++++++++++++++++++++ 6 files changed, 115 insertions(+), 49 deletions(-) create mode 100755 scripts/free-disk-space.sh diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 866ed107e3..20632f6361 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,6 +37,10 @@ jobs: run: sudo ifconfig lo0 alias 127.0.0.3 - uses: actions/checkout@v3 + + - name: Free Disk Space + if: matrix.target.os == 'ubuntu-latest' + run: ./scripts/free-disk-space.sh - name: Install OpenSSL if: matrix.target.os == 'windows-latest' diff --git a/actix-server/CHANGES.md b/actix-server/CHANGES.md index 1f9f7b3c66..ffe3f2a1a0 100644 --- a/actix-server/CHANGES.md +++ b/actix-server/CHANGES.md @@ -1,22 +1,22 @@ # Changes -## Unreleased - 2023-xx-xx +## Unreleased - Add support for MultiPath TCP (MPTCP) with `MpTcp` enum and `ServerBuilder::mptcp()` method. - Minimum supported Rust version (MSRV) is now 1.65. -## 2.2.0 - 2022-12-21 +## 2.2.0 - Minimum supported Rust version (MSRV) is now 1.59. - Update `tokio-uring` dependency to `0.4`. [#473] [#473]: https://github.com/actix/actix-net/pull/473 -## 2.1.1 - 2022-03-09 +## 2.1.1 - No significant changes since `2.1.0`. -## 2.1.0 - 2022-03-08 +## 2.1.0 - Update `tokio-uring` dependency to `0.3`. [#448] - Logs emitted now use the `tracing` crate with `log` compatibility. [#448] @@ -25,27 +25,27 @@ [#443]: https://github.com/actix/actix-net/pull/443 [#448]: https://github.com/actix/actix-net/pull/448 -## 2.0.0 - 2022-01-19 +## 2.0.0 - No significant changes since `2.0.0-rc.4`. -## 2.0.0-rc.4 - 2022-01-12 +## 2.0.0-rc.4 - Update `tokio-uring` dependency to `0.2`. [#436] [#436]: https://github.com/actix/actix-net/pull/436 -## 2.0.0-rc.3 - 2021-12-31 +## 2.0.0-rc.3 - No significant changes since `2.0.0-rc.2`. -## 2.0.0-rc.2 - 2021-12-27 +## 2.0.0-rc.2 - Simplify `TestServer`. [#431] [#431]: https://github.com/actix/actix-net/pull/431 -## 2.0.0-rc.1 - 2021-12-05 +## 2.0.0-rc.1 - Hide implementation details of `Server`. [#424] - `Server` now runs only after awaiting it. [#425] @@ -53,19 +53,19 @@ [#424]: https://github.com/actix/actix-net/pull/424 [#425]: https://github.com/actix/actix-net/pull/425 -## 2.0.0-beta.9 - 2021-11-15 +## 2.0.0-beta.9 - Restore `Arbiter` support lost in `beta.8`. [#417] [#417]: https://github.com/actix/actix-net/pull/417 -## 2.0.0-beta.8 - 2021-11-05 _(YANKED)_ +## 2.0.0-beta.8 - Fix non-unix signal handler. [#410] [#410]: https://github.com/actix/actix-net/pull/410 -## 2.0.0-beta.7 - 2021-11-05 _(YANKED)_ +## 2.0.0-beta.7 - Server can be started in regular Tokio runtime. [#408] - Expose new `Server` type whose `Future` impl resolves when server stops. [#408] @@ -78,7 +78,7 @@ [#407]: https://github.com/actix/actix-net/pull/407 [#408]: https://github.com/actix/actix-net/pull/408 -## 2.0.0-beta.6 - 2021-10-11 +## 2.0.0-beta.6 - Add experimental (semver-exempt) `io-uring` feature for enabling async file I/O on linux. [#374] - Server no long listens to `SIGHUP` signal. Previously, the received was not used but did block subsequent exit signals from working. [#389] @@ -89,19 +89,19 @@ [#349]: https://github.com/actix/actix-net/pull/349 [#389]: https://github.com/actix/actix-net/pull/389 -## 2.0.0-beta.5 - 2021-04-20 +## 2.0.0-beta.5 - Server shutdown notifies all workers to exit regardless if shutdown is graceful. This causes all workers to shutdown immediately in force shutdown case. [#333] [#333]: https://github.com/actix/actix-net/pull/333 -## 2.0.0-beta.4 - 2021-04-01 +## 2.0.0-beta.4 - Prevent panic when `shutdown_timeout` is very large. [f9262db] [f9262db]: https://github.com/actix/actix-net/commit/f9262db -## 2.0.0-beta.3 - 2021-02-06 +## 2.0.0-beta.3 - Hidden `ServerBuilder::start` method has been removed. Use `ServerBuilder::run`. [#246] - Add retry for EINTR signal (`io::Interrupted`) in `Accept`'s poll loop. [#264] @@ -113,13 +113,13 @@ [#265]: https://github.com/actix/actix-net/pull/265 [#273]: https://github.com/actix/actix-net/pull/273 -## 2.0.0-beta.2 - 2021-01-03 +## 2.0.0-beta.2 - Merge `actix-testing` to `actix-server` as `test_server` mod. [#242] [#242]: https://github.com/actix/actix-net/pull/242 -## 2.0.0-beta.1 - 2020-12-28 +## 2.0.0-beta.1 - Added explicit info log message on accept queue pause. [#215] - Prevent double registration of sockets when back-pressure is resolved. [#223] @@ -134,127 +134,127 @@ [#223]: https://github.com/actix/actix-net/pull/223 [#239]: https://github.com/actix/actix-net/pull/239 -## 1.0.4 - 2020-09-12 +## 1.0.4 - Update actix-codec to 0.3.0. - Workers must be greater than 0. [#167] [#167]: https://github.com/actix/actix-net/pull/167 -## 1.0.3 - 2020-05-19 +## 1.0.3 - Replace deprecated `net2` crate with `socket2` [#140] [#140]: https://github.com/actix/actix-net/pull/140 -## 1.0.2 - 2020-02-26 +## 1.0.2 - Avoid error by calling `reregister()` on Windows [#103] [#103]: https://github.com/actix/actix-net/pull/103 -## 1.0.1 - 2019-12-29 +## 1.0.1 - Rename `.start()` method to `.run()` -## 1.0.0 - 2019-12-11 +## 1.0.0 - Use actix-net releases -## 1.0.0-alpha.4 - 2019-12-08 +## 1.0.0-alpha.4 - Use actix-service 1.0.0-alpha.4 -## 1.0.0-alpha.3 - 2019-12-07 +## 1.0.0-alpha.3 - Migrate to tokio 0.2 - Fix compilation on non-unix platforms - Better handling server configuration -## 1.0.0-alpha.2 - 2019-12-02 +## 1.0.0-alpha.2 - Simplify server service (remove actix-server-config) - Allow to wait on `Server` until server stops -## 0.8.0-alpha.1 - 2019-11-22 +## 0.8.0-alpha.1 - Migrate to `std::future` -## 0.7.0 - 2019-10-04 +## 0.7.0 - Update `rustls` to 0.16 - Minimum required Rust version upped to 1.37.0 -## 0.6.1 - 2019-09-25 +## 0.6.1 - Add UDS listening support to `ServerBuilder` -## 0.6.0 - 2019-07-18 +## 0.6.0 - Support Unix domain sockets #3 -## 0.5.1 - 2019-05-18 +## 0.5.1 - ServerBuilder::shutdown_timeout() accepts u64 -## 0.5.0 - 2019-05-12 +## 0.5.0 - Add `Debug` impl for `SslError` - Derive debug for `Server` and `ServerCommand` - Upgrade to actix-service 0.4 -## 0.4.3 - 2019-04-16 +## 0.4.3 - Re-export `IoStream` trait - Depend on `ssl` and `rust-tls` features from actix-server-config -## 0.4.2 - 2019-03-30 +## 0.4.2 - Fix SIGINT force shutdown -## 0.4.1 - 2019-03-14 +## 0.4.1 - `SystemRuntime::on_start()` - allow to run future before server service initialization -## 0.4.0 - 2019-03-12 +## 0.4.0 - Use `ServerConfig` for service factory - Wrap tcp socket to `Io` type - Upgrade actix-service -## 0.3.1 - 2019-03-04 +## 0.3.1 - Add `ServerBuilder::maxconnrate` sets the maximum per-worker number of concurrent connections - Add helper ssl error `SslError` - Rename `StreamServiceFactory` to `ServiceFactory` - Deprecate `StreamServiceFactory` -## 0.3.0 - 2019-03-02 +## 0.3.0 - Use new `NewService` trait -## 0.2.1 - 2019-02-09 +## 0.2.1 - Drop service response -## 0.2.0 - 2019-02-01 +## 0.2.0 - Migrate to actix-service 0.2 - Updated rustls dependency -## 0.1.3 - 2018-12-21 +## 0.1.3 - Fix max concurrent connections handling -## 0.1.2 - 2018-12-12 +## 0.1.2 - rename ServiceConfig::rt() to ServiceConfig::apply() - Fix back-pressure for concurrent ssl handshakes -## 0.1.1 - 2018-12-11 +## 0.1.1 - Fix signal handling on windows -## 0.1.0 - 2018-12-09 +## 0.1.0 - Move server to separate crate diff --git a/actix-server/Cargo.toml b/actix-server/Cargo.toml index 62f1310794..53f54f7950 100755 --- a/actix-server/Cargo.toml +++ b/actix-server/Cargo.toml @@ -27,7 +27,6 @@ actix-utils = "3" futures-core = { version = "0.3.17", default-features = false, features = ["alloc"] } futures-util = { version = "0.3.17", default-features = false, features = ["alloc"] } mio = { version = "0.8", features = ["os-poll", "net"] } -num_cpus = "1.13" socket2 = "0.5" tokio = { version = "1.23.1", features = ["sync"] } tracing = { version = "0.1.30", default-features = false, features = ["log"] } diff --git a/actix-server/src/builder.rs b/actix-server/src/builder.rs index e1d3a2d328..42addafce3 100644 --- a/actix-server/src/builder.rs +++ b/actix-server/src/builder.rs @@ -1,4 +1,4 @@ -use std::{io, time::Duration}; +use std::{io, num::NonZeroUsize, time::Duration}; use actix_rt::net::TcpStream; use tokio::sync::mpsc::{unbounded_channel, UnboundedReceiver, UnboundedSender}; @@ -55,7 +55,7 @@ impl ServerBuilder { let (cmd_tx, cmd_rx) = unbounded_channel(); ServerBuilder { - threads: num_cpus::get_physical(), + threads: std::thread::available_parallelism().map_or(2, NonZeroUsize::get), token: 0, factories: Vec::new(), sockets: Vec::new(), @@ -76,6 +76,12 @@ impl ServerBuilder { /// The default worker count is the number of physical CPU cores available. If your benchmark /// testing indicates that simultaneous multi-threading is beneficial to your app, you can use /// the [`num_cpus`] crate to acquire the _logical_ core count instead. + /// + /// # Panics + /// + /// Panics if `num` is 0. + /// + /// [`num_cpus`]: https://docs.rs/num_cpus pub fn workers(mut self, num: usize) -> Self { assert_ne!(num, 0, "workers must be greater than 0"); self.threads = num; diff --git a/actix-server/src/worker.rs b/actix-server/src/worker.rs index a9ae97a004..d72cd2dbf1 100644 --- a/actix-server/src/worker.rs +++ b/actix-server/src/worker.rs @@ -1,6 +1,7 @@ use std::{ future::Future, io, mem, + num::NonZeroUsize, pin::Pin, rc::Rc, sync::{ @@ -249,8 +250,11 @@ pub(crate) struct ServerWorkerConfig { impl Default for ServerWorkerConfig { fn default() -> Self { - // 512 is the default max blocking thread count of tokio runtime. - let max_blocking_threads = std::cmp::max(512 / num_cpus::get_physical(), 1); + let parallelism = std::thread::available_parallelism().map_or(2, NonZeroUsize::get); + + // 512 is the default max blocking thread count of a Tokio runtime. + let max_blocking_threads = std::cmp::max(512 / parallelism, 1); + Self { shutdown_timeout: Duration::from_secs(30), max_blocking_threads, diff --git a/scripts/free-disk-space.sh b/scripts/free-disk-space.sh new file mode 100755 index 0000000000..2946cfcf6e --- /dev/null +++ b/scripts/free-disk-space.sh @@ -0,0 +1,53 @@ +#!/usr/bin/env bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# The Azure provided machines typically have the following disk allocation: +# Total space: 85GB +# Allocated: 67 GB +# Free: 17 GB +# This script frees up 28 GB of disk space by deleting unneeded packages and +# large directories. +# The Flink end to end tests download and generate more than 17 GB of files, +# causing unpredictable behavior and build failures. + +echo "==============================================================================" +echo "Freeing up disk space on CI system" +echo "==============================================================================" + +echo "Listing 100 largest packages" +dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -n | tail -n 100 +df -h + +echo "Removing large packages" +sudo apt-get remove -y '^dotnet-.*' +sudo apt-get remove -y 'php.*' +sudo apt-get remove -y '^mongodb-.*' +sudo apt-get remove -y '^mysql-.*' +sudo apt-get remove -y azure-cli google-cloud-sdk hhvm google-chrome-stable firefox powershell mono-devel libgl1-mesa-dri +sudo apt-get autoremove -y +sudo apt-get clean +df -h + +echo "Removing large directories" +sudo rm -rf /usr/share/dotnet/ +sudo rm -rf /usr/local/graalvm/ +sudo rm -rf /usr/local/.ghcup/ +sudo rm -rf /usr/local/share/powershell +sudo rm -rf /usr/local/share/chromium +sudo rm -rf /usr/local/lib/android +sudo rm -rf /usr/local/lib/node_modules +df -h