misc: merge main (#581)

* feat(blog): add missing sqlx migration code to auth blog post (#408)

* feat(blog): add missing sqlx migration code

* fix: nit

* misc: 0.7.0 (#407)

* misc: drop patches

* infra: pin postgres to version 14

* refactor: fix thruster example app name

* infra: db_fqdn fix

* tests: warp hello world

* fix(cargo-shuttle): prevent crash when config owned by root (#409)

* fix: use correct timeout start point (#410)

* fix: use correct timeout start point

* tweak health check frequency

* fmt

* feat(deployer): implement container memory limits (#411)

* feat(deployer): implement container memory limits

* test: fix warp integration test

* bug: `transport error` when trying to connect to provisioner (#416)

* bug: keep provisioner connections alive

* refactor: reduce scope provisioner client is running

* feat: gateway admin revive (#412)

* feat: gateway admin command (revive)

* fmt

* clippy

* refactor: revive deployers using GatewayService

* tests: add ContextArgs

* refactor: simplify passing around of fqdn

* tests: update test archive

* refactor: remove stray exec.rs file

* refactor: unused is_error()

Co-authored-by: chesedo <pieter@chesedo.me>

* bug: timeout curl health check on deployer (#415)

* Article/beta article (#420)

* updated docs url

* beta article w/ damiens corrections

* Feat(www): shuttle beta signup (#421)

* feat(www): add beta signup form

* feat: add formspree endpoint

* feat: remove socials footer from beta page

* feat: add dummy text above sign up form

* feat: placeholder gif

* feat: align dummy text left

* feat: remove gif, add text

* feat: update beta page text

* feat: update the rest of the site

* bug: Fix thruster postgres example (#414)

* feat: shell completions (#343)

* bug: package Secrets.toml (#422)

* bug: package Secrets.toml

* refactor: clippy suggestion

* bug: big archives being cut off at 32 768 bytes (#423)

* refactor: set chunked header

* refactor: don't send stream across threads

Doing this is causing us to loose everything after 32 768 bytes. Don't
know the reason why 🤷

* refactor: fix health check

* refactor: remove unused use

* chore: v0.7.1 (#424)

* chore: v0.7.1

* chore: resources v0.7.1

* docs: v0.7.1

* tests: v0.7.1

* tests: fix warp test

* tests: fix warp test

* tests: fix secrets e2e test

* ci: remove hard coded DD env

* misc: unintended shell changes

* refactor: cargo check suggestion

* refactor: provide better context for errors (#430)

* refactor: do our own health checks on deployer containers (#427)

* refactor: do our own health checks on deployer containers

We are doing this because the docker health checks eats up 25% of a
single CPU core and we are spinning up many of these containers.

https://www.reddit.com/r/docker/comments/b68r53/healtchecks_add_high_cpu_load/
moby/moby#39102

* refactor: clippy suggestion

* bug: clear build folder before extracting (#428)

* refactor: make sure extract directory is created

* tests: test build folder is cleared

* bug: clear the build directory

* added some images (#435)

* feat(www): beta blog updates (#434)

* feat: add captioned image component (#440)

* feat: add captioned image component

* feat: update images

* feat: add header caption

* fix: quotes in header

* www: post small tweaks (#439)

* a few tweaks to the article

* feat: update gif

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* feat: replace cursed gif (#441)

* Feat/update contributing (#426)

* feat(docs): update contributing.md

* feat(e2e): make e2e admin user unique

* feat(e2e): clean up e2e test projects on test complete

* feat(e2e): don't fail on superuser conflict

* feat: add section about contributing to docs

* feat: remove unwrap, add comment to drop impl

* tests: update e2e test key

Co-authored-by: chesedo <pieter@chesedo.me>

* chore: v0.7.2 (#442)

* chore: v0.7.2

* chore: resources v0.7.2

* docs: v0.7.2

* misc: v0.7.2

* misc: script to deploy all examples

* misc: targets for publishing crates

* misc: remove old scripts

* misc: instructions for next steps

* misc: get Cargo.lock changes

* misc: update .PHONY

* misc: show usage

* feat: link the tracing spans between services (#445)

* ci: DD environments

* refactor: try lowering broadcast channel for quicker feedback

* refactor: propagate tracing across threads in deployer

* refactor: propagate tracing between gateway and deployer

* refactor: trace account name

* refactor: associate project with each container

* feat: trace and propagate proxy in gateway

* feat: gateway record project

* refactor: tracing use 'error' for errors

* refactor: fix comment

* refactor: drop recording unused field

* Add docker-compose extra flags param in Makefile (#446)

* Add docker-compose extra flags param

* rename to avoid confusion

* misc: restructure repo (#453)

* misc: only run e2e on production

* misc: move www to shuttle-hq/www repo

* misc: move examples to shuttle-hq/examples repo

* refactor: update links to examples repo

* ci: remove checking fmt of examples

* ci: checkout submodules

* refactor: tf files have been moved to shuttle-hq/terraform-aws-shuttle

* fix: wrap around common::ProjectName for parsing (#451)

* fix: gateway state drifts, health checks and project recreation (#447)

* misc: add more helpful flags to Makefile

* misc: remove old migrator (#463)

* feat: add account_tier column (#458)

* feat: prefetch shuttle-service crates (#461)

* feat: prefetch shuttle-service crates

* refactor: add comment to prepare.sh files

* Feat: revive via gateway endpoint (#460)

* feat: revive via gateway api initial commit

* feat: cleanup, document gateway testing

* refactor: use post for revive endpoint

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: send task to global queue

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: create a new admin cli binary crate (#462)

* feat: init shuttle-admin

* feat: add args

* feat: simple client

* feat: read api_key

* refactor: hook it all together

* refactor: switch to post

* fix: broken link (#467)

* fix: add timeout to health checks (#468)

* fix: add timeout to health checks

* Parameterise the timeout

* rollback on inadvertent circleci change

* misc: log out a warning when a task has been running for a long time

* longer timeouts

* Address review

* Fmt

* feat(gateway): add custom domains table and routing (#465)

* WIP feat: count recent start events before restart (#469)

* feat: count recent start events before restart

* feat: try_collect and handle error in start counter

* fix: clippy

* refactor: gateway clippy

* feat: make deployer only answer its own project (#466)

* feat: make deployer only answer its own project

* bug: use correct project name

* refactor: make backwards compatible

* Fixed Links in Readme (#477)

* refactor: base client error off response status code (#470)

* refactor: base client error off response status code

* Update common/src/models/error.rs

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>

* refactor: use tracing

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>

* feat: verify project exists before sending destroy task (#474)

* feat: add a custom domains admin route (#473)

Co-authored-by: Alex Krantz <alex@krantz.dev>

* bug: deployer freezes (#478)

* bug: reduce spawning to have deployer lock up less

* refactor: don't return cargo logs

* ci: green (#482)

* tests: allow longer time for build

* refactor: trim dependencies on persist

* refactor: clippy suggestion

* feat: TLS acceptor with SNI resolver (#471)

* fix: custom domain routing (#484)

* refactor: more metrics (#475)

* feat: add more metric dimensions to deployer

* feat: add more metric dimensions to gateway

* refactor: common metrics code

* refactor: forward account name

* refactor: add backend feature to deployer

* refactor: standardize naming

* refactor: cargo sort

* misc: configurable deployment tags (#486)

* feat: gateway restores removed containers (#485)

* fix: backend bumps and hot fixes (#487)

* Feature/support actix web (#491)

* Support actix-web: impl Service

* Support actix-web: cli + boilerplate code

* Support actix-web: add framework to doc comment

* Support actix-web: add e2e test

* Support actix-web: fmt

* Support actix-web: sorting toml deps

* Support actix-web: add actix-web to ci

* Support actix-web: formatted boilerplate code for actix-web hello world example

* Support actix-web: formatted boilerplate code for actix-web hello world example
Support actix-web: update cargo lock zstd dep

* Support actix-web: simplify example

* Support actix-web: add test, change example dependency

* fix: e2e test assert, reset example module

Co-authored-by: maksim <maksim@blockchaincmdt.io>
Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* Doc: Fix command to prime database with docker-compose (#502)

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

* Doc: Improve contributing documentation (#499)

In the doc it mentions to go to subfolder of examples but it was not
there :).

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>

* feat: static file support for a single folder (#501)

* feat: static-folder resource

* refactor: make local client work

* refactor: make local deployer work

* feat: storage_manager

* refactor: comments

* refactor: remove unwraps and expects

* refactor: update tests

* refactor: clippy suggestion

* refactor: update version

* refactor: update readme

* refactor: update comment

* refactor: change public to static

* ci: add static-folder

* refactor: code fixes

* refactor: update tests

* refactor: update description

Co-authored-by: Damien <damienbroka@mailbox.org>

Co-authored-by: Damien <damienbroka@mailbox.org>

* ci: build binary (#483)

* ci: build binary

* ci: make tar archive

* ci: make GH draft release with artifacts

* ci: add aarch64

* ci: get version from tag

* ci: rename to linux builds

* ci: add windows

* ci: add macos

* ci: test all

* ci: put all the artifacts under a command

* ci: only on tagged

* ci: restore old

* ci: fix windows

* ci: put archive in separate folder

* ci: switch to tar

* ci: restore old

* refactor: better `cargo-binstall` support

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

* refactor: binary name

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>

* feat: bump rust to 1.64, bump dependencies (#495)

* feat: bump rust to 1.64, correct contributing.md bug

* feat: bump service deps, change example branch

* fix: pin chrono to 4.22

* chore: upgrade workspace dependencies

* feat: remove aws-smithy-types

ref: smithy-lang/smithy-rs#1760

* feat: upgrade axum to 0.6 in common

* feat: upgrade to axum 0.6.0 in deployer

* feat: upgrade gateway to axum 0.6.0

* feat: upgrde sqlx to 0.6.2 in gateway

* feat: bump chrono to 0.4.23

* feat: replace deprecated chrono functions in deployer

* feat: bump chrono in common

* feat: ignore actix integration test

* feat: implement new state extractor in gateway

* feat: interactive project initialization (#498)

* feat: bump pinned rust version to 1.65 (#504)

* feat: bump pinned rust version to 1.65

 lockfile update is from binary dist PR #483

* ci: install newer protoc

* fix: special module name warning

* fix: clippy

* feat: install newer protoc in dockerfile as well

* misc: env updates (#509)

* ci: update toolchain for binaries

* misc: update dev environment

* misc: remove patches from Cargo.lock

* feat: make the folder configurable (#508)

* feat: make the folder configurable

* Update resources/static-folder/README.md

Co-authored-by: Damien <damienbroka@mailbox.org>

Co-authored-by: Damien <damienbroka@mailbox.org>

* feat: implement workspace inheritance (#506)

* refactor: switch away from cargo package (#507)

* feat: add spinner wait for `project new` and `project status --follow` (#503)

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>
Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* fix: capitalise correctly (#511)

* fix: make nice (#512)

* feat: find (soon to be) invalid project names (#479)

* feat: find (soon to be) invalid project names

* refactor: move logic to admin client

* refactor: missed axum 0.6 update (#513)

* feat: build tests in release profile, limit build workers (#514)

* feat: build tests in release profile, limit build workers

* fix: typo in manifests

* feat: create `init` project from correct dir (#518)

* misc: interactive init gif (#519)

* Feat/set examples submodule to main (#520)

* feat: revert examples submodule back to main

* feat: un-ignore actix tests

* chore: 0.8.0 (#521)

* chore: bump examples (#522)

* bug: hacking static folders (#524)

* fix: actix integration with state (#523)

* feat(gateway,deployer): add more tracing events (#500)

* feat(deployer): add more tracing events

* feat(gateway): add more tracing events

* feat: canonicalize before trace (#531)

* feat: 'clean' subcommand (#530)

* feat: 'clean' subcommand

* refactor: output cleaning is done

* Feat/set cpu limit (#529)

* feat: remove redundant actix worker limit

* feat: remove redundant build job limit

* feat: set cpu limit for deployer container

* fix: limit actix worker and build jobs to 4

* feat: add panamax for mirroring crates.io (#525)

* feat: build queue (#532)

* feat: per-project parallelism (#533)

* feat: temp validation of project name in gateway (#534)

* fix(deployer): keep Cargo.lock between deployments (#517)

* ci: remove build and push req on build binaries (#535)

* refactor: don't crash when failing to release slot (#536)

* refactor: release build slot parse type correctly (#538)

* refactor: remove prefetch (#539)

* feat: add cron job for syncing mirror (#537)

* chore: bump cargo-shuttle to 0.8.1 (#540)

* chore: bump cargo-shuttle to 0.8.1

* chore: cargo.lock

* test: fixes (#554)

* tests: custom domains

* tests: end to end

* Refactor: remove deprecated auth command (#550)

* refactor: remove deprecated auth command

* docs: update main readme

* ci: add Makefile command for Windows to convert .sh files to LF format (#555)

* Add documentation for unblocking Windows CRLF

+ Add documentation for unblocking CRLF issues in Windows using git commands

* refactor: footnote for git link, remove extra newlines

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* Fix: dependencies compiled with incompatible versions of rustc (#545)

* feat: disable honor_rust_version

* feat: add rust toolchain override to deployer

* feat: remove rust-toolchain from deployer

* feat: add RUSTUP_TOOLCHAIN env to deployer container

* feat: get currently installed rustc version

* fix: better naming

* feat: set containerfile rust version via makefile

also set the RUSTUP_TOOLCHAIN env var via the dockerfile, and pass it to deployers

* refactor: remove unused dep

Co-authored-by: Pieter <pieter@chesedo.me>

* refactor: remove toolchain env from deployer config

Co-authored-by: Pieter <pieter@chesedo.me>

* feat: create subcommand to list all projects of calling account (#553)

* feat: create subcommand to list all projects of calling account

* fix: filter query by account name

* tests: iter_user_projects_detailed

Co-authored-by: chesedo <pieter@chesedo.me>

* bug: deployer drifting state (#548)

* ci: fix bin builds (#546)

* ci: comment out other jobs than linux bins

* ci: try without tag

* feat: set git tag with git describe

* feat: echo tag in CI for debugging

* fix: set tag earlier for testing

* ci: echo artifacts directory

* ci: pass tag via parameters

* ci: move artifacts in target sub directory

* ci: create artifacts subdir

* ci: test draft release

* ci: authenticate ghr

* ci: try with symlink

* ci: revert symlink, cp bins from subdirs before release

* ci: flatten artifacts dir before releasing

* ci: set-tag command

* ci: fix invalid config

* fix: invalid config

* ci: test if artifacts flattening command works for win bin

* ci: enable linux bin build too

* ci: remove environment field

* ci: check artifacts dir structure

* ci: tweak ghr command and verify tag is set

* ci: include tag in make-artifact env

* fix: invalid command

* fix: incorrect path to bash.env

* ci: set tag in env in preceding step

* ci: clean up, add comments

* fix: rm path

* ci: re-enable all workflows, remove tag filters

* fix: formatting

* update contributing (#556)

Make docker compose commands consistent

* feat: Support Poise (#560)

* feat: impl Service for Poise

* feat(poise): add to ci

* doc(poise): add link to poise example

* feat(poise): add poise to cargo shuttle init

Co-authored-by: oddgrd <29732646+oddgrd@users.noreply.github.com>

* feat: add flag for router IP local run (#565)

* feat: add flag for router IP local run

* refactor: amend ipaddr setting, local run init fn

* refactor: delete unnecessary dependency

Forgot to delete a dependency and it looks like I can't squash commits, whoops 🤦

* Revert "refactor: delete unnecessary dependency"

This reverts commit 1794cc6.

* refactor: remove unnecessary dep (amendment)
Re-pushing this amendment but without the git diff. Probably should have changed that before submitting initially.

* clippy

* amendment from clippy warning (needless borrow)

* refactor: apply cargo fmt diffs

* refactor: return URL from run test setup fn
+ cargo_shuttle_run() now returns a string containing the full base URL instead of port
+ Relevant changes have been made to tests to account for this

* refactor: change flag name to be 'external'

Changed as per discussion on the PR.

* refactor: use passed args directly for struct

* test: make sure tests align to proper folders

Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>

* bug: no networks (#541)

* refactor: connect to user network on startup

* feat: revive containers not connected to the user network

* refactor: attach network first

* refactor: revive created containers

* feat: attaching state

* refactor: resetting some code

Signed-off-by: Federico Guerinoni <guerinoni.federico@gmail.com>
Co-authored-by: Oddbjørn Grødem <29732646+oddgrd@users.noreply.github.com>
Co-authored-by: Alexander Krantz <alex@krantz.dev>
Co-authored-by: Damien <git@damien.sh>
Co-authored-by: Ivan <ivancernja@gmail.com>
Co-authored-by: Peter Mertz <peter.s.mertz@gmail.com>
Co-authored-by: XyLyXyRR <39663597+XyLyXyRR@users.noreply.github.com>
Co-authored-by: Damien <damienbroka@mailbox.org>
Co-authored-by: Nereuxofficial <37740907+Nereuxofficial@users.noreply.github.com>
Co-authored-by: Maxim <59533214+biryukovmaxim@users.noreply.github.com>
Co-authored-by: maksim <maksim@blockchaincmdt.io>
Co-authored-by: Federico Guerinoni <41150432+guerinoni@users.noreply.github.com>
Co-authored-by: Isaiah Gamble <77396670+tsar-boomba@users.noreply.github.com>
Co-authored-by: Stijn ("stain") Seghers <stijnseghers@gmail.com>
Co-authored-by: Joshua Mo <102877324+joshua-mo-143@users.noreply.github.com>
Co-authored-by: Emma <85257928+emmakuen@users.noreply.github.com>
Co-authored-by: Oleks Gnatovskyi <22867443+alekspickle@users.noreply.github.com>
Co-authored-by: Anafabula <57800226+Anafabula@users.noreply.github.com>

.circleci/config.yml

@@ -148,7 +148,7 @@ jobs:
       - restore-cargo-cache
       - install-protoc
       - run: cargo fmt --all --check
-      # TODO: this is incompatible with workspace inheritance, uncomment when 
+      # TODO: this is incompatible with workspace inheritance, uncomment when
       # https://github.com/DevinR528/cargo-sort/pull/29 is merged
       # - run: cargo install cargo-sort
       # - run: cargo sort --check --workspace
@@ -185,7 +185,7 @@ jobs:
       - install-protoc
       - apply-patches
       - run: cargo fmt --all --check --manifest-path << parameters.path >>/Cargo.toml
-      # TODO: this is incompatible with workspace inheritance, uncomment when 
+      # TODO: this is incompatible with workspace inheritance, uncomment when
       # https://github.com/DevinR528/cargo-sort/pull/29 is merged
       # - run: cargo install cargo-sort
       # - run: cargo sort --check << parameters.path >>

cargo-shuttle/src/lib.rs

@@ -27,7 +27,6 @@ use futures::{StreamExt, TryFutureExt};
 use git2::{Repository, StatusOptions};
 use ignore::overrides::OverrideBuilder;
 use ignore::WalkBuilder;
-use provisioner_server::LocalProvisioner;
 use shuttle_common::models::{project, secret};
 use shuttle_service::loader::{build_crate, Runtime};
 use std::fmt::Write;
@@ -38,6 +37,7 @@ use uuid::Uuid;
 
 use crate::args::{DeploymentCommand, ProjectCommand};
 use crate::client::Client;
+use crate::provisioner_server::LocalProvisioner;
 
 pub struct Shuttle {
     ctx: RequestContext,
@@ -479,6 +479,7 @@ impl Shuttle {
 
         trace!(response = ?response,  "client response: ");
 
+        // TODO: move to runtime
         let addr = if run_args.external {
             std::net::IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0))
         } else {

deployer/tests/deploy_layer/bind-panic/Cargo.toml

@@ -11,4 +11,4 @@ crate-type = ["cdylib"]
 [workspace]
 
 [dependencies]
-shuttle-service = { version = "0.8.0", features = ["codegen"] }
+shuttle-service = "0.8.0"

deployer/tests/deploy_layer/main-panic/Cargo.toml

@@ -11,4 +11,4 @@ crate-type = ["cdylib"]
 [workspace]
 
 [dependencies]
-shuttle-service = { version = "0.8.0", features = ["codegen"] }
+shuttle-service = "0.8.0"

deployer/tests/deploy_layer/self-stop/Cargo.toml

@@ -11,4 +11,4 @@ crate-type = ["cdylib"]
 [workspace]
 
 [dependencies]
-shuttle-service = { version = "0.8.0", features = ["codegen"] }
+shuttle-service = "0.8.0"

deployer/tests/deploy_layer/sleep-async/Cargo.toml

@@ -12,4 +12,4 @@ crate-type = ["cdylib"]
 
 [dependencies]
 tokio = { version = "1.0", features = ["time"]}
-shuttle-service = { version = "0.8.0", features = ["codegen"] }
+shuttle-service = "0.8.0"

deployer/tests/resources/tests-fail/Cargo.toml

@@ -10,4 +10,4 @@ crate-type = ["cdylib"]
 
 [dependencies]
 rocket = "0.5.0-rc.2"
-shuttle-service = { path = "../../../../service", features = ["codegen", "web-rocket"] }
+shuttle-service = { path = "../../../../service", features = ["web-rocket"] }

deployer/tests/resources/tests-pass/Cargo.toml

@@ -10,4 +10,4 @@ crate-type = ["cdylib"]
 
 [dependencies]
 rocket = "0.5.0-rc.2"
-shuttle-service = { path = "../../../../service", features = ["codegen", "web-rocket"] }
+shuttle-service = { path = "../../../../service", features = ["web-rocket"] }

service/tests/resources/sleep-async/Cargo.toml

@@ -10,4 +10,4 @@ crate-type = ["cdylib"]
 
 [dependencies]
 tokio = { version = "1.22.0", features = ["time"] }
-shuttle-service = { path = "../../../", features = ["codegen"] }
+shuttle-service = { path = "../../../" }

shell.nix

@@ -12,8 +12,8 @@ in
     ];
     buildInputs = with nixpkgs; [
       ((rustChannelOf{ channel = "1.65.0"; }).rust.override {
-        targets = ["wasm32-wasi"];
         extensions = ["rust-src"];
+        targets = ["wasm32-wasi"];
       })
       cargo-watch
       terraform