Restructure project as Cargo workspace and drop direct dependency on a HTTP server, and also CF Worker (wasm) runtime

[dotansimha]

Closes #131 #39 #98

• Setup Cargo workspace
• Split benches into a crate
• Move core impl into a crate (engine)
• Create new bin entry point (conductor crate)
• Split config into a crate
• Improve specific types and structs and make it more common and easier to consume (specifically Request/Response/Body)
• Remove dependency for HTTP/runtime from the core package
• Switch entry point to use actix-web
• Rust 1.74 for CI
• Fix CI
• Added CF Worker runtime

[github-actions]

🐋 This PR was built and pushed to the following Docker images:

Docker Bake metadata

{
"conductor": {
  "containerimage.config.digest": "sha256:89060319fcf765939468f865fb03d91c5117a9b8437ce093e8a77797091ca0a5",
  "containerimage.descriptor": {
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "digest": "sha256:9f1423b1835a2b37efe46358d0bcb7cd3c6223aa2444f5bbfd67a33470a5f270",
    "size": 901,
    "platform": {
      "architecture": "amd64",
      "os": "linux"
    }
  },
  "containerimage.digest": "sha256:9f1423b1835a2b37efe46358d0bcb7cd3c6223aa2444f5bbfd67a33470a5f270",
  "image.name": "ghcr.io/the-guild-org/conductor-t2/conductor:bf6ece8f9a84928ae4198b4b1118008aff1153c6"
}
}

[YassinEldeeb]

when I tried running locally, I got a few of this warning:

2023-11-19T01:32:35.930689Z  WARN actix_router::resource: Tail segments must have names. Consider `.../{tail}*`. This may become a panic in the future.
2023-11-19T01:32:35.930853Z  WARN actix_router::resource: Tail segments must have names. Consider `.../{tail}*`. This may become a panic in the future.
2023-11-19T01:32:35.930998Z  WARN actix_router::resource: Tail segments must have names. Consider `.../{tail}*`. This may become a panic in the future.

In Actix-web, when defining a route that captures all the remaining path segments, we should give a name to these segments, which are referred in the error as the "tail" segment. The warning suggests that we're currently using a route that captures all the remaining segments but we don't name them.

Also, the warning mentions that this might become a panic in future versions of Actix-web.

btw, Actix-web lets us catch the trailing segments and access it in the handler like this:

async fn handler(
    req: HttpRequest,
    body: Bytes,
    route_data: web::Data<ConductorGatewayRouteData>,
    gw: web::Data<ConductorGateway>,
    path: web::Path<(String,)>, // Optionally added to capture the tail segment, in case we ever need it
) -> impl Responder {
    // ....
    let tail_segment = &path.0;

Suggested change

	.route("/.*", web::route().to(handler));
	.route("/{tail:.*}", web::route().to(handler));

[YassinEldeeb]

Update: using named segments path breaks path matching, visiting: /graphql and /test doesn't work after the switch, maybe we should wait for a fix...I tried debugging it for some time, but I can't seem to get it to work without anonymous segments matching.

[YassinEldeeb]

I believe strip default value is false

Suggested change

strip: false

[YassinEldeeb]

I'm not sure if async_runtime abstraction is needed in its own crate, it's very minimal.
and it's only to encounter cross platform runtime differences with wasm.

Also, it does handle reqwest client initiation, so the name async_runtime is kind of confusing.

But, also at the same time, I appreciate separating it, since this can grow much larger in the future as we progress, it also eases testing cross-platform compatibility, so how about the name wasm_polyfills instead?

and when it comes to usage, to be like this:

-let fetcher = create_http_client().build().unwrap();
+let fetcher = wasm_polyfills::create_http_client().build().unwrap();

So, we eliminate a few jumps to imports & implementation, when reading inline code usage.

[YassinEldeeb]

I had no idea { workspace = true } was added to infer workspace root dependencies' versions
Last time I checked, it was an abandoned RFC 😄
rust-lang/rfcs#2906

This is really cool! Good to know!

[YassinEldeeb]

I believe we can separate the workspace instead of:

crates
├── async_runtime
├── benches
├── cloudflare_worker
├── common
├── conductor
├── config
└── engine

to:

bins
├── conductor
└── cloudflare_worker
libs
├── async_runtime
├── benches
├── common
├── config
└── engine

So, that there is separation between what is executable and what is code specific like libraries. what do you think? 👀

[YassinEldeeb]

I feel like from_yaml isn't clear when interpreted inline at the first glance,
how about we rename it to parse_config_from_yaml(..)?

[YassinEldeeb]

since we added this utility, we can also use it above, and extract one for json too like this:

pub fn parse_config_from_yaml(contents: &str) -> Result<ConductorConfig, serde_yaml::Error> {
    serde_yaml::from_str::<ConductorConfig>(contents)
}

pub fn parse_config_from_json(contents: &str) -> Result<ConductorConfig, serde_json::Error> {
    serde_json::from_str::<ConductorConfig>(contents)
}

and above, we use those utilities too:

match path.extension() {
    Some(ext) => match ext.to_str() {
        Some("json") => parse_config_from_json(&contents).expect("Failed to parse config file"),
        Some("yaml") | Some("yml") => {
            parse_config_from_yaml(&contents).expect("Failed to parse config file")
        }
    }
}  

[YassinEldeeb]

I wonder where are we using this? 🤔

[YassinEldeeb]

YESSS! Can't wait to see how that impacts the bench numbers

[YassinEldeeb]

That is Incredible!
It's super clean 💯

[YassinEldeeb]

Since, the feedback is opinionated, and nothing is really crucial.
I've applied the feedback here #136, so you can checkout to the branch, try it,
and get a feel for it, and decide if you like to merge it or not.

We can merge this now!