Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ jobs:

- name: Wait for active storage server to start
run: |
until curl -if http://localhost:8080/.well-known/s3-active-storage-schema; do
until curl -if http://localhost:8080/.well-known/reductionist-schema; do
sleep 1;
done

Expand All @@ -74,7 +74,7 @@ jobs:
run: pytest -s > artifacts/pytest.log

- name: Get active storage logs
run: docker logs s3-active-storage > artifacts/s3-active-storage.log
run: docker logs reductionist > artifacts/reductionist.log
if: always()

- name: Upload artifacts
Expand Down
84 changes: 42 additions & 42 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[package]
name = "s3-active-storage"
name = "reductionist"
version = "0.2.0"
edition = "2021"
# Due to AWS SDK.
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,5 +21,5 @@ FROM debian:bullseye-slim
RUN apt update \
&& apt install -y --no-install-recommends ca-certificates \
&& update-ca-certificates
COPY --from=builder /usr/local/cargo/bin/s3-active-storage /usr/local/bin/s3-active-storage
CMD ["s3-active-storage"]
COPY --from=builder /usr/local/cargo/bin/reductionist /usr/local/bin/reductionist
CMD ["reductionist"]
12 changes: 6 additions & 6 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
.PHONY: build
build:
@docker buildx build -t s3-active-storage .
@docker buildx build -t reductionist .

.PHONY: test
test:
@docker buildx build --build-arg PROFILE=dev --target builder -t s3-active-storage-test .
@docker run --rm s3-active-storage-test cargo check --color always
@docker run --rm s3-active-storage-test cargo test --color always
@docker buildx build --build-arg PROFILE=dev --target builder -t reductionist-test .
@docker run --rm reductionist-test cargo check --color always
@docker run --rm reductionist-test cargo test --color always

.PHONY: run
run:
@docker run -it --detach --rm --net=host --name s3-active-storage s3-active-storage
@docker run -it --detach --rm --net=host --name reductionist reductionist

.PHONY: stop
stop:
@docker stop s3-active-storage
@docker stop reductionist
32 changes: 16 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# S3 Active Storage
# Reductionist

This project implements simple reductions on S3 objects containing numeric binary data.
By implementing these reductions in the storage system the volume of data that needs to be
Expand All @@ -10,7 +10,7 @@ and is done in collaboration with the
[University of Reading](http://www.reading.ac.uk/).

This is a performant implementation of the active storage server.
The original Python functional prototype is available [here](https://github.com/stackhpc/s3-active-storage).
The original Python functional prototype is available [here](https://github.com/stackhpc/reductionist).

## Concepts

Expand Down Expand Up @@ -104,17 +104,17 @@ In particular, the following are known limitations which we intend to address:

## Running

There are various ways to run the S3 Active Storage server.
There are various ways to run the Reductionist server.

### Running in a container

The simplest method is to run it in a container using a pre-built image:

```sh
docker run -it --detach --rm --net=host --name s3-active-storage ghcr.io/stackhpc/s3-active-storage-rs:latest
docker run -it --detach --rm --net=host --name reductionist ghcr.io/stackhpc/reductionist-rs:latest
```

Images are published to [GitHub Container Registry](https://github.com/stackhpc/s3-active-storage-rs/pkgs/container/s3-active-storage-rs) when the project is released.
Images are published to [GitHub Container Registry](https://github.com/stackhpc/reductionist-rs/pkgs/container/reductionist-rs) when the project is released.
The `latest` tag corresponds to the most recent release, or you can use a specific release e.g. `0.1.0`.

This method does not require access to the source code.
Expand All @@ -125,15 +125,15 @@ If you need to use unreleased changes, but still want to run in a container, it
First, clone this repository:

```sh
git clone https://github.com/stackhpc/s3-active-storage-rs.git
cd s3-active-storage-rs
git clone https://github.com/stackhpc/reductionist-rs.git
cd reductionist-rs
```

```sh
make build
```

The image will be tagged as `s3-active-storage`.
The image will be tagged as `reductionist`.
The image may be pushed to a registry, or deployed locally.

```sh
Expand All @@ -142,7 +142,7 @@ make run

## Build

If you prefer not to run the S3 Active Storage server in a container, it will be necessary to build a binary.
If you prefer not to run the Reductionist server in a container, it will be necessary to build a binary.
Building locally may also be preferable during development to take advantage of incremental compilation.

### Prerequisites
Expand All @@ -157,8 +157,8 @@ See the [Rust book](https://doc.rust-lang.org/book/ch01-01-installation.html) fo
First, clone this repository:

```sh
git clone https://github.com/stackhpc/s3-active-storage-rs.git
cd s3-active-storage-rs
git clone https://github.com/stackhpc/reductionist-rs.git
cd reductionist-rs
```

Next, use Cargo to build the package:
Expand All @@ -182,7 +182,7 @@ cargo install --path . --locked
Then run:

```sh
s3-active-storage
reductionist
```

## Testing
Expand Down Expand Up @@ -223,7 +223,7 @@ python ./scripts/upload_sample_data.py

### Compliance test suite

Proxy functionality can be tested using the [S3 active storage compliance suite](https://github.com/stackhpc/s3-active-storage-compliance-suite).
Proxy functionality can be tested using the [S3 active storage compliance suite](https://github.com/stackhpc/reductionist-compliance-suite).

### Making requests to active storage endpoints

Expand Down Expand Up @@ -251,15 +251,15 @@ venv/bin/python ./scripts/client.py sum --server http://localhost:8080 --source
## Documentation

The source code is documented using [rustdoc](https://doc.rust-lang.org/rustdoc/what-is-rustdoc.html).
Currently the `s3-active-storage` crate is not uploaded to https://crates.io, so we do not benefit from hosted documentation on https://docs.rs.
Currently the `reductionist` crate is not uploaded to https://crates.io, so we do not benefit from hosted documentation on https://docs.rs.
It is however possible to build the documentation locally:

```sh
cargo doc
```

Cargo builds documentation for the `s3-active-storage` crate and all of its dependencies.
The resulting documentation is available under `target/doc`, and may be viewed in a web browser using file:///path/to/s3-active-storage/target/doc/s3-active-storage/index.html.
Cargo builds documentation for the `reductionist` crate and all of its dependencies.
The resulting documentation is available under `target/doc`, and may be viewed in a web browser using file:///path/to/reductionist/target/doc/reductionist/index.html.

## Contributing

Expand Down
2 changes: 1 addition & 1 deletion benches/shuffle.rs
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
/// Benchmarks for the byte shuffle filter implementation.
use criterion::{black_box, criterion_group, criterion_main, Criterion};
use s3_active_storage::filters::shuffle;
use reductionist::filters::shuffle;

fn criterion_benchmark(c: &mut Criterion) {
for size_k in [64, 256, 1024] {
Expand Down
4 changes: 2 additions & 2 deletions src/app.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,13 @@ fn router() -> Router {
}

Router::new()
.route("/.well-known/s3-active-storage-schema", get(schema))
.route("/.well-known/reductionist-schema", get(schema))
.route("/metrics", get(metrics_handler))
.nest("/v1", v1())
.route_layer(middleware::from_fn(track_metrics))
}

/// S3 Active Storage Server Service type alias
/// Reductionist Server Service type alias
///
/// This type implements [tower_service::Service].
// FIXME: The Service type should be some form of tower_service::Service, but couldn't find the
Expand Down
18 changes: 9 additions & 9 deletions src/cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,34 @@

use clap::Parser;

/// S3 Active Storage Proxy command line interface
/// Reductionist command line interface
#[derive(Debug, Parser)]
pub struct CommandLineArgs {
/// The IP address on which the proxy should listen
#[arg(long, default_value = "0.0.0.0", env = "S3_ACTIVE_STORAGE_HOST")]
#[arg(long, default_value = "0.0.0.0", env = "REDUCTIONIST_HOST")]
pub host: String,
/// The port to which the proxy should bind
#[arg(long, default_value_t = 8080, env = "S3_ACTIVE_STORAGE_PORT")]
#[arg(long, default_value_t = 8080, env = "REDUCTIONIST_PORT")]
pub port: u16,
/// Flag indicating whether HTTPS should be used
#[arg(long, default_value_t = false, env = "S3_ACTIVE_STORAGE_HTTPS")]
#[arg(long, default_value_t = false, env = "REDUCTIONIST_HTTPS")]
pub https: bool,
/// Path to the certificate file to be used for HTTPS encryption
#[arg(
long,
default_value = "~/.config/s3-active-storage/certs/cert.pem",
env = "S3_ACTIVE_STORAGE_CERT_FILE"
default_value = "~/.config/reductionist/certs/cert.pem",
env = "REDUCTIONIST_CERT_FILE"
)]
pub cert_file: String,
/// Path to the key file to be used for HTTPS encryption
#[arg(
long,
default_value = "~/.config/s3-active-storage/certs/key.pem",
env = "S3_ACTIVE_STORAGE_KEY_FILE"
default_value = "~/.config/reductionist/certs/key.pem",
env = "REDUCTIONIST_KEY_FILE"
)]
pub key_file: String,
/// Maximum time in seconds to wait for operations to complete upon receiving `ctrl+c` signal.
#[arg(long, default_value_t = 60, env = "S3_ACTIVE_STORAGE_SHUTDOWN_TIMEOUT")]
#[arg(long, default_value_t = 60, env = "REDUCTIONIST_SHUTDOWN_TIMEOUT")]
pub graceful_shutdown_timeout: u64,
}

Expand Down
2 changes: 1 addition & 1 deletion src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
//!
//! This is a performant implementation of the Active Storage Server.
//! The original Python functional prototype is available
//! [here](https://github.com/stackhpc/s3-active-storage).
//! [here](https://github.com/stackhpc/reductionist).
//!
//! The Active Storage Server is built on top of a number of open source components.
//!
Expand Down
12 changes: 6 additions & 6 deletions src/main.rs
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
//! This file defines the s3-active-storage binary entry point.
//! This file defines the reductionist binary entry point.

use s3_active_storage::app;
use s3_active_storage::cli;
use s3_active_storage::metrics;
use s3_active_storage::server;
use s3_active_storage::tracing;
use reductionist::app;
use reductionist::cli;
use reductionist::metrics;
use reductionist::server;
use reductionist::tracing;

/// Application entry point
#[tokio::main]
Expand Down
2 changes: 1 addition & 1 deletion src/server.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use axum_server::{tls_rustls::RustlsConfig, Handle};
use expanduser::expanduser;
use tokio::signal;

/// Serve the S3 Active Storage service
/// Serve the Reductionist service
///
/// # Arguments
///
Expand Down
2 changes: 1 addition & 1 deletion src/tracing.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ pub fn init_tracing() {
tracing_subscriber::registry()
.with(
tracing_subscriber::EnvFilter::try_from_default_env()
.unwrap_or_else(|_| "s3_active_storage=debug,tower_http=debug".into()),
.unwrap_or_else(|_| "reductionist=debug,tower_http=debug".into()),
)
.with(tracing_subscriber::fmt::layer())
.init();
Expand Down