diff --git a/docs/libraries/workflow/DESIGN.md b/docs/libraries/workflow/DESIGN.md new file mode 100644 index 000000000..cf4c86881 --- /dev/null +++ b/docs/libraries/workflow/DESIGN.md @@ -0,0 +1,5 @@ +# Design + +## Hierarchy + +TODO diff --git a/docs/libraries/workflow/GLOSSARY.md b/docs/libraries/workflow/GLOSSARY.md index c03544218..dcc7e51d0 100644 --- a/docs/libraries/workflow/GLOSSARY.md +++ b/docs/libraries/workflow/GLOSSARY.md @@ -15,8 +15,9 @@ A collection of registered workflows. This is solely used for the worker to fetc A series of fallible executions of code (also known as activities), signal listeners, signal transmitters, or sub workflow triggers. -Workflows can be though of as a list of tasks. The code defining a workflow only specifies what items should -be ran; There is no complex logic (e.g. database queries) running within the top level of the workflow. +Workflows can be though of as an outline or a list of tasks. The code defining a workflow only specifies what +items should be ran; There is no complex logic (e.g. database queries) running within the top level of the +workflow. Upon an activity failure, workflow code can be reran without duplicate side effects because activities are cached and re-read after they succeed. @@ -27,6 +28,11 @@ A block of code that can fail. This cannot trigger other workflows or activities Activities are retried by workflows when they fail or replayed when they succeed but a later part of the workflow fails. +When choosing between a workflow and an activity: + +- Choose a workflow when there are multiple steps that need to be individually retried upon failure. +- Choose an activity when there is only one chunk of retryable code that needs to be executed. + ## Operation Effectively a native rust function. Can fail or not fail. Used for widely used operations like fetching a @@ -51,6 +57,10 @@ this signal for it to be picked up, otherwise it will stay in the database indef workflow. Signals do not have a response; another signal must be sent back from the workflow and listened to by the sender. +### Differences between message + +Signals are like messages that can only be consumed by workflows and can only be consumed once. + ## Tagged Signal Same as a signal except it is sent with a JSON blob as its "tags" instead of to a specific workflow. Any @@ -65,6 +75,29 @@ See [the signals document](./SIGNALS.md). A "one of" for signal listening. Allows for listening to multiple signals at once and receiving the first one that gets sent. +## Message + +A payload that can be sent out of a workflow. Includes a JSON blob for tags which can be subscribed to with a +subscription. + +### Differences between signal + +Messages are like signals that can be only consumed by non workflows and can be consumed by multiple +listeners. + +## Subscription + +An entity that waits for messages with the same (not a superset/subset) tags as itself. Upon receiving a +message, the message will be returned and the developer can choose to continue to listen for more messages. + +## Tail + +Reads the last message without waiting. If none exists (all previous messages expired), `None` is returned. + +## Tail w/ Anchor + +Reads the earliest message after the given anchor timestamp or waits for one to be published if none exist. + ## Workflow Event An action that gets executed in a workflow. An event can be a: diff --git a/docs/libraries/workflow/GOTCHAS.md b/docs/libraries/workflow/GOTCHAS.md new file mode 100644 index 000000000..4528dad44 --- /dev/null +++ b/docs/libraries/workflow/GOTCHAS.md @@ -0,0 +1,20 @@ +# Gotchas + +## Timestamps + +Use timestamps with care when passing them between activity inputs/outputs. Because activity inputs need to be +consistent for replays, use `util::timestamp::now()` only within activities and not workflow bodies. + +If you need a timestamp in a workflow body, use `ctx.create_ts()` for the creation of the workflow. Using +`ctx.ts()` is also inconsistent because it marks the start of the current workflow run (which is different +between replays). + +If you need a consistent current timestamp, create a new activity that just returns `util::timestamp::now()`. +This will be the current timestamp on the first execution of the activity and won't change on replay. + +> **When an activity's input doesn't produce the same hash as the first time it was executed (i.e. its input +> changed), the entire workflow will error with "History Diverged" and will not restart.** + +## Randomly generated content + +Randomly generated content like UUIDs should be placed in activities for consistent history. diff --git a/docs/libraries/workflow/SIGNALS.md b/docs/libraries/workflow/SIGNALS.md deleted file mode 100644 index cc71587d3..000000000 --- a/docs/libraries/workflow/SIGNALS.md +++ /dev/null @@ -1,7 +0,0 @@ -# Signals - -## Tagged signals - -Tagged signals are consumed on a first-come-first-serve basis because a single signal being consumed by more -than one workflow is not a supported design pattern. To work around this, consume the signal by a workflow -then publish multiple signals from that workflow. diff --git a/docs/libraries/workflow/SIGNALS_AND_MESSAGES.md b/docs/libraries/workflow/SIGNALS_AND_MESSAGES.md new file mode 100644 index 000000000..49eb2ecbf --- /dev/null +++ b/docs/libraries/workflow/SIGNALS_AND_MESSAGES.md @@ -0,0 +1,29 @@ +# Signals + +## Tagged signals + +Tagged signals are consumed on a first-come-first-serve basis because a single signal being consumed by more +than one workflow is not a supported design pattern. To work around this, consume the signal by a workflow +then publish multiple signals from that workflow. + +# Choosing Between Signals and Messages + +> **Note**: non-workflow ecosystem is API layer, standalone, operations, old workers + +## Signal + +- Sending data from the non-workflow ecosystem to the workflow ecosystem +- Sending data from the workflow ecosystem to somewhere else in the workflow ecosystem + +## Message + +- Sending data from the workflow ecosystem to the non-workflow ecosystem + +## Both Signals and Messages + +Sometimes you may need to listen for a particular event in the workflow system and the non-workflow ecosystem. +In this case you can publish both a signal and a message (you can derive `signal` and `message` on the same +struct to make this easier). Just remember: signals can only be consumed once. + +Both messages and signals are meant to be payloads with a specific recipient. They are not meant to be +published without an intended target (i.e. any listener can consume). diff --git a/fern/definition/admin/clusters/common.yml b/fern/definition/admin/clusters/common.yml index b879f6e39..cd5d8ce66 100644 --- a/fern/definition/admin/clusters/common.yml +++ b/fern/definition/admin/clusters/common.yml @@ -51,7 +51,7 @@ types: Server: properties: server_id: uuid - public_ip: string + public_ip: optional PoolUpdate: properties: diff --git a/lib/bolt/cli/src/commands/cluster/mod.rs b/lib/bolt/cli/src/commands/cluster/mod.rs index c730a7028..33a6e7470 100644 --- a/lib/bolt/cli/src/commands/cluster/mod.rs +++ b/lib/bolt/cli/src/commands/cluster/mod.rs @@ -234,7 +234,8 @@ impl SubCommand { let server_ips = servers .servers .iter() - .map(|x| x.public_ip.as_str()) + .filter_map(|x| x.public_ip.as_ref()) + .map(|x| x.as_str()) .collect::>(); // SSH in to servers diff --git a/lib/bolt/config/src/ns.rs b/lib/bolt/config/src/ns.rs index 2e3dcbea3..6c1458bba 100644 --- a/lib/bolt/config/src/ns.rs +++ b/lib/bolt/config/src/ns.rs @@ -614,9 +614,6 @@ pub struct NsfwCheck { pub struct Provisioning { /// Default cluster. pub cluster: Option, - /// Whether or not to send a taint message in the next cluster update. - #[serde(default)] - pub taint: bool, /// How many empty job servers to have at all times. Used in the simple provisioning algorithm on Rivet /// Enterprise. #[serde(default = "default_job_server_provision_margin")] diff --git a/lib/bolt/core/src/context/service.rs b/lib/bolt/core/src/context/service.rs index d1bf9b184..7557cdeb4 100644 --- a/lib/bolt/core/src/context/service.rs +++ b/lib/bolt/core/src/context/service.rs @@ -1053,14 +1053,6 @@ impl ServiceContextData { "RIVET_DEFAULT_CLUSTER_CONFIG".into(), serde_json::to_string(&provisioning.cluster)?, ); - env.insert( - "RIVET_TAINT_DEFAULT_CLUSTER".into(), - if provisioning.taint { - "1".to_string() - } else { - "0".to_string() - }, - ); } if self.depends_on_provision_margin() { diff --git a/lib/bolt/core/src/tasks/gen.rs b/lib/bolt/core/src/tasks/gen.rs index 9511dba8d..eac2916df 100644 --- a/lib/bolt/core/src/tasks/gen.rs +++ b/lib/bolt/core/src/tasks/gen.rs @@ -162,12 +162,6 @@ async fn generate_root(path: &Path) { } } } - - // Utils lib - let util_path = pkg.path().join("util"); - if fs::metadata(&util_path).await.is_ok() { - set_license(&util_path.join("Cargo.toml")).await; - } } } diff --git a/lib/chirp-workflow/core/src/compat.rs b/lib/chirp-workflow/core/src/compat.rs index 2674be7f3..8f24b69fd 100644 --- a/lib/chirp-workflow/core/src/compat.rs +++ b/lib/chirp-workflow/core/src/compat.rs @@ -1,14 +1,20 @@ // Forwards compatibility from old operation ctx to new workflows -use std::{fmt::Debug, time::Duration}; +use std::fmt::Debug; use global_error::prelude::*; use serde::Serialize; use uuid::Uuid; use crate::{ - ctx::api::WORKFLOW_TIMEOUT, DatabaseHandle, DatabasePostgres, Operation, OperationCtx, - OperationInput, Signal, Workflow, WorkflowError, WorkflowInput, + ctx::{ + api::WORKFLOW_TIMEOUT, + message::{MessageCtx, SubscriptionHandle}, + workflow::SUB_WORKFLOW_RETRY, + }, + message::Message, + DatabaseHandle, DatabasePostgres, Operation, OperationCtx, OperationInput, Signal, Workflow, + WorkflowError, WorkflowInput, }; pub async fn dispatch_workflow( @@ -83,30 +89,32 @@ where } /// Wait for a given workflow to complete. -/// **IMPORTANT:** Has no timeout. +/// 60 second timeout. pub async fn wait_for_workflow( ctx: &rivet_operation::OperationContext, workflow_id: Uuid, ) -> GlobalResult { tracing::info!(name=W::NAME, id=?workflow_id, "waiting for workflow"); - let period = Duration::from_millis(50); - let mut interval = tokio::time::interval(period); - loop { - interval.tick().await; - - // Check if state finished - let workflow = db_from_ctx(ctx) - .await? - .get_workflow(workflow_id) - .await - .map_err(GlobalError::raw)? - .ok_or(WorkflowError::WorkflowNotFound) - .map_err(GlobalError::raw)?; - if let Some(output) = workflow.parse_output::().map_err(GlobalError::raw)? { - return Ok(output); + tokio::time::timeout(WORKFLOW_TIMEOUT, async move { + let mut interval = tokio::time::interval(SUB_WORKFLOW_RETRY); + loop { + interval.tick().await; + + // Check if state finished + let workflow = db_from_ctx(ctx) + .await? + .get_workflow(workflow_id) + .await + .map_err(GlobalError::raw)? + .ok_or(WorkflowError::WorkflowNotFound) + .map_err(GlobalError::raw)?; + if let Some(output) = workflow.parse_output::().map_err(GlobalError::raw)? { + return Ok(output); + } } - } + }) + .await? } /// Dispatch a new workflow and wait for it to complete. Has a 60s timeout. @@ -121,11 +129,7 @@ where { let workflow_id = dispatch_workflow(ctx, input).await?; - tokio::time::timeout( - WORKFLOW_TIMEOUT, - wait_for_workflow::(ctx, workflow_id), - ) - .await? + wait_for_workflow::(ctx, workflow_id).await } /// Dispatch a new workflow and wait for it to complete. Has a 60s timeout. @@ -141,11 +145,7 @@ where { let workflow_id = dispatch_tagged_workflow(ctx, tags, input).await?; - tokio::time::timeout( - WORKFLOW_TIMEOUT, - wait_for_workflow::(ctx, workflow_id), - ) - .await? + wait_for_workflow::(ctx, workflow_id).await } pub async fn signal( @@ -226,6 +226,21 @@ where .map_err(GlobalError::raw) } +pub async fn subscribe( + ctx: &rivet_operation::OperationContext, + tags: &serde_json::Value, +) -> GlobalResult> +where + M: Message, + B: Debug + Clone, +{ + let msg_ctx = MessageCtx::new(ctx.conn(), ctx.req_id(), ctx.ray_id()) + .await + .map_err(GlobalError::raw)?; + + msg_ctx.subscribe::(tags).await.map_err(GlobalError::raw) +} + // Get crdb pool as a trait object async fn db_from_ctx( ctx: &rivet_operation::OperationContext, diff --git a/lib/chirp-workflow/core/src/ctx/activity.rs b/lib/chirp-workflow/core/src/ctx/activity.rs index ec488fa22..6a4119616 100644 --- a/lib/chirp-workflow/core/src/ctx/activity.rs +++ b/lib/chirp-workflow/core/src/ctx/activity.rs @@ -93,6 +93,10 @@ impl ActivityCtx { self.name } + pub fn workflow_id(&self) -> Uuid { + self.workflow_id + } + pub fn req_id(&self) -> Uuid { self.op_ctx.req_id() } diff --git a/lib/chirp-workflow/core/src/ctx/api.rs b/lib/chirp-workflow/core/src/ctx/api.rs index 222872433..1c91b83f7 100644 --- a/lib/chirp-workflow/core/src/ctx/api.rs +++ b/lib/chirp-workflow/core/src/ctx/api.rs @@ -7,8 +7,9 @@ use uuid::Uuid; use crate::{ ctx::{ - message::{SubscriptionHandle, TailAnchor, TailAnchorResponse}, - MessageCtx, OperationCtx, + message::{MessageCtx, SubscriptionHandle, TailAnchor, TailAnchorResponse}, + operation::OperationCtx, + workflow::SUB_WORKFLOW_RETRY, }, error::WorkflowResult, message::{Message, ReceivedMessage}, @@ -123,30 +124,32 @@ impl ApiCtx { } /// Wait for a given workflow to complete. - /// **IMPORTANT:** Has no timeout. + /// 60 second timeout. pub async fn wait_for_workflow( &self, workflow_id: Uuid, ) -> GlobalResult { tracing::info!(name=W::NAME, id=?workflow_id, "waiting for workflow"); - let period = Duration::from_millis(50); - let mut interval = tokio::time::interval(period); - loop { - interval.tick().await; - - // Check if state finished - let workflow = self - .db - .get_workflow(workflow_id) - .await - .map_err(GlobalError::raw)? - .ok_or(WorkflowError::WorkflowNotFound) - .map_err(GlobalError::raw)?; - if let Some(output) = workflow.parse_output::().map_err(GlobalError::raw)? { - return Ok(output); + tokio::time::timeout(WORKFLOW_TIMEOUT, async move { + let mut interval = tokio::time::interval(SUB_WORKFLOW_RETRY); + loop { + interval.tick().await; + + // Check if state finished + let workflow = self + .db + .get_workflow(workflow_id) + .await + .map_err(GlobalError::raw)? + .ok_or(WorkflowError::WorkflowNotFound) + .map_err(GlobalError::raw)?; + if let Some(output) = workflow.parse_output::().map_err(GlobalError::raw)? { + return Ok(output); + } } - } + }) + .await? } /// Dispatch a new workflow and wait for it to complete. Has a 60s timeout. @@ -159,12 +162,7 @@ impl ApiCtx { ::Workflow: Workflow, { let workflow_id = self.dispatch_workflow(input).await?; - - tokio::time::timeout( - WORKFLOW_TIMEOUT, - self.wait_for_workflow::(workflow_id), - ) - .await? + self.wait_for_workflow::(workflow_id).await } /// Dispatch a new workflow with tags and wait for it to complete. Has a 60s timeout. @@ -178,12 +176,7 @@ impl ApiCtx { ::Workflow: Workflow, { let workflow_id = self.dispatch_tagged_workflow(tags, input).await?; - - tokio::time::timeout( - WORKFLOW_TIMEOUT, - self.wait_for_workflow::(workflow_id), - ) - .await? + self.wait_for_workflow::(workflow_id).await } pub async fn signal( @@ -326,10 +319,6 @@ impl ApiCtx { self.ts.saturating_sub(self.op_ctx.req_ts()) } - // pub fn perf(&self) -> &chirp_perf::PerfCtx { - // self.conn.perf() - // } - pub fn trace(&self) -> &[chirp_client::TraceEntry] { self.conn.trace() } diff --git a/lib/chirp-workflow/core/src/ctx/mod.rs b/lib/chirp-workflow/core/src/ctx/mod.rs index 00dc5bab1..68c2f2848 100644 --- a/lib/chirp-workflow/core/src/ctx/mod.rs +++ b/lib/chirp-workflow/core/src/ctx/mod.rs @@ -2,13 +2,13 @@ mod activity; pub(crate) mod api; pub mod message; mod operation; +mod standalone; mod test; -mod workflow; +pub(crate) mod workflow; pub use activity::ActivityCtx; pub use api::ApiCtx; pub use message::MessageCtx; pub use operation::OperationCtx; +pub use standalone::StandaloneCtx; pub use test::TestCtx; pub use workflow::WorkflowCtx; - -// TODO: StandaloneCtx diff --git a/lib/chirp-workflow/core/src/ctx/operation.rs b/lib/chirp-workflow/core/src/ctx/operation.rs index 4094d77a2..104c683cb 100644 --- a/lib/chirp-workflow/core/src/ctx/operation.rs +++ b/lib/chirp-workflow/core/src/ctx/operation.rs @@ -1,9 +1,11 @@ use global_error::{GlobalError, GlobalResult}; use rivet_pools::prelude::*; +use serde::Serialize; use uuid::Uuid; -use crate::{DatabaseHandle, Operation, OperationInput, WorkflowError}; +use crate::{signal::Signal, DatabaseHandle, Operation, OperationInput, WorkflowError}; +#[derive(Clone)] pub struct OperationCtx { ray_id: Uuid, name: &'static str, @@ -75,6 +77,50 @@ impl OperationCtx { .map_err(WorkflowError::OperationFailure) .map_err(GlobalError::raw) } + + pub async fn signal( + &self, + workflow_id: Uuid, + input: T, + ) -> GlobalResult { + let signal_id = Uuid::new_v4(); + + tracing::info!(name=%T::NAME, %workflow_id, %signal_id, "dispatching signal"); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeSignalBody) + .map_err(GlobalError::raw)?; + + self.db + .publish_signal(self.ray_id, workflow_id, signal_id, T::NAME, input_val) + .await + .map_err(GlobalError::raw)?; + + Ok(signal_id) + } + + pub async fn tagged_signal( + &self, + tags: &serde_json::Value, + input: T, + ) -> GlobalResult { + let signal_id = Uuid::new_v4(); + + tracing::info!(name=%T::NAME, ?tags, %signal_id, "dispatching tagged signal"); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeSignalBody) + .map_err(GlobalError::raw)?; + + self.db + .publish_tagged_signal(self.ray_id, tags, signal_id, T::NAME, input_val) + .await + .map_err(GlobalError::raw)?; + + Ok(signal_id) + } } impl OperationCtx { diff --git a/lib/chirp-workflow/core/src/ctx/standalone.rs b/lib/chirp-workflow/core/src/ctx/standalone.rs new file mode 100644 index 000000000..f3bd819f4 --- /dev/null +++ b/lib/chirp-workflow/core/src/ctx/standalone.rs @@ -0,0 +1,375 @@ +use global_error::{GlobalError, GlobalResult}; +use rivet_pools::prelude::*; +use serde::Serialize; +use uuid::Uuid; + +use crate::{ + ctx::{ + api::WORKFLOW_TIMEOUT, + message::{SubscriptionHandle, TailAnchor, TailAnchorResponse}, + workflow::SUB_WORKFLOW_RETRY, + MessageCtx, OperationCtx, + }, + error::WorkflowResult, + message::{Message, ReceivedMessage}, + DatabaseHandle, Operation, OperationInput, Signal, Workflow, WorkflowError, WorkflowInput, +}; + +#[derive(Clone)] +pub struct StandaloneCtx { + ray_id: Uuid, + name: &'static str, + ts: i64, + + db: DatabaseHandle, + + conn: rivet_connection::Connection, + msg_ctx: MessageCtx, + + // Backwards compatibility + op_ctx: rivet_operation::OperationContext<()>, +} + +impl StandaloneCtx { + pub async fn new( + db: DatabaseHandle, + conn: rivet_connection::Connection, + name: &'static str, + ) -> WorkflowResult { + let req_id = Uuid::new_v4(); + let ray_id = Uuid::new_v4(); + let ts = rivet_util::timestamp::now(); + + let op_ctx = rivet_operation::OperationContext::new( + name.to_string(), + std::time::Duration::from_secs(60), + conn.clone(), + req_id, + ray_id, + ts, + ts, + (), + ); + + let msg_ctx = MessageCtx::new(&conn, req_id, ray_id).await?; + + Ok(StandaloneCtx { + ray_id, + name, + ts, + db, + conn, + op_ctx, + msg_ctx, + }) + } +} + +impl StandaloneCtx { + pub async fn dispatch_workflow(&self, input: I) -> GlobalResult + where + I: WorkflowInput, + ::Workflow: Workflow, + { + let name = I::Workflow::NAME; + + tracing::info!(%name, ?input, "dispatching workflow"); + + let id = Uuid::new_v4(); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeWorkflowOutput) + .map_err(GlobalError::raw)?; + + self.db + .dispatch_workflow(self.ray_id, id, &name, None, input_val) + .await + .map_err(GlobalError::raw)?; + + tracing::info!(%name, ?id, "workflow dispatched"); + + Ok(id) + } + + pub async fn dispatch_tagged_workflow( + &self, + tags: &serde_json::Value, + input: I, + ) -> GlobalResult + where + I: WorkflowInput, + ::Workflow: Workflow, + { + let name = I::Workflow::NAME; + + tracing::info!(%name, ?tags, ?input, "dispatching tagged workflow"); + + let id = Uuid::new_v4(); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeWorkflowOutput) + .map_err(GlobalError::raw)?; + + self.db + .dispatch_workflow(self.ray_id, id, &name, Some(tags), input_val) + .await + .map_err(GlobalError::raw)?; + + tracing::info!(%name, ?id, "workflow dispatched"); + + Ok(id) + } + + /// Wait for a given workflow to complete. + /// 60 second timeout. + pub async fn wait_for_workflow( + &self, + workflow_id: Uuid, + ) -> GlobalResult { + tracing::info!(name=W::NAME, id=?workflow_id, "waiting for workflow"); + + tokio::time::timeout(WORKFLOW_TIMEOUT, async move { + let mut interval = tokio::time::interval(SUB_WORKFLOW_RETRY); + loop { + interval.tick().await; + + // Check if state finished + let workflow = self + .db + .get_workflow(workflow_id) + .await + .map_err(GlobalError::raw)? + .ok_or(WorkflowError::WorkflowNotFound) + .map_err(GlobalError::raw)?; + if let Some(output) = workflow.parse_output::().map_err(GlobalError::raw)? { + return Ok(output); + } + } + }) + .await? + } + + /// Dispatch a new workflow and wait for it to complete. Has a 60s timeout. + pub async fn workflow( + &self, + input: I, + ) -> GlobalResult<<::Workflow as Workflow>::Output> + where + I: WorkflowInput, + ::Workflow: Workflow, + { + let workflow_id = self.dispatch_workflow(input).await?; + self.wait_for_workflow::(workflow_id).await + } + + /// Dispatch a new workflow with tags and wait for it to complete. Has a 60s timeout. + pub async fn tagged_workflow( + &self, + tags: &serde_json::Value, + input: I, + ) -> GlobalResult<<::Workflow as Workflow>::Output> + where + I: WorkflowInput, + ::Workflow: Workflow, + { + let workflow_id = self.dispatch_tagged_workflow(tags, input).await?; + self.wait_for_workflow::(workflow_id).await + } + + pub async fn signal( + &self, + workflow_id: Uuid, + input: T, + ) -> GlobalResult { + let signal_id = Uuid::new_v4(); + + tracing::info!(name=%T::NAME, %workflow_id, %signal_id, "dispatching signal"); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeSignalBody) + .map_err(GlobalError::raw)?; + + self.db + .publish_signal(self.ray_id, workflow_id, signal_id, T::NAME, input_val) + .await + .map_err(GlobalError::raw)?; + + Ok(signal_id) + } + + pub async fn tagged_signal( + &self, + tags: &serde_json::Value, + input: T, + ) -> GlobalResult { + let signal_id = Uuid::new_v4(); + + tracing::info!(name=%T::NAME, ?tags, %signal_id, "dispatching tagged signal"); + + // Serialize input + let input_val = serde_json::to_value(input) + .map_err(WorkflowError::SerializeSignalBody) + .map_err(GlobalError::raw)?; + + self.db + .publish_tagged_signal(self.ray_id, tags, signal_id, T::NAME, input_val) + .await + .map_err(GlobalError::raw)?; + + Ok(signal_id) + } + + pub async fn op( + &self, + input: I, + ) -> GlobalResult<<::Operation as Operation>::Output> + where + I: OperationInput, + ::Operation: Operation, + { + let ctx = OperationCtx::new( + self.db.clone(), + &self.conn, + self.ray_id, + self.op_ctx.req_ts(), + false, + I::Operation::NAME, + ); + + I::Operation::run(&ctx, &input) + .await + .map_err(WorkflowError::OperationFailure) + .map_err(GlobalError::raw) + } + + pub async fn subscribe( + &self, + tags: &serde_json::Value, + ) -> GlobalResult> + where + M: Message, + { + self.msg_ctx + .subscribe::(tags) + .await + .map_err(GlobalError::raw) + } + + pub async fn tail_read( + &self, + tags: serde_json::Value, + ) -> GlobalResult>> + where + M: Message, + { + self.msg_ctx + .tail_read::(tags) + .await + .map_err(GlobalError::raw) + } + + pub async fn tail_anchor( + &self, + tags: serde_json::Value, + anchor: &TailAnchor, + ) -> GlobalResult> + where + M: Message, + { + self.msg_ctx + .tail_anchor::(tags, anchor) + .await + .map_err(GlobalError::raw) + } +} + +impl StandaloneCtx { + pub fn name(&self) -> &str { + self.name + } + + // pub fn timeout(&self) -> Duration { + // self.timeout + // } + + pub fn req_id(&self) -> Uuid { + self.op_ctx.req_id() + } + + pub fn ray_id(&self) -> Uuid { + self.ray_id + } + + /// Timestamp at which the request started. + pub fn ts(&self) -> i64 { + self.ts + } + + /// Timestamp at which the request was published. + pub fn req_ts(&self) -> i64 { + self.op_ctx.req_ts() + } + + /// Time between when the timestamp was processed and when it was published. + pub fn req_dt(&self) -> i64 { + self.ts.saturating_sub(self.op_ctx.req_ts()) + } + + pub fn trace(&self) -> &[chirp_client::TraceEntry] { + self.conn.trace() + } + + pub fn test(&self) -> bool { + self.trace() + .iter() + .any(|x| x.run_context == chirp_client::RunContext::Test as i32) + } + + pub fn chirp(&self) -> &chirp_client::Client { + self.conn.chirp() + } + + pub fn cache(&self) -> rivet_cache::RequestConfig { + self.conn.cache() + } + + pub fn cache_handle(&self) -> rivet_cache::Cache { + self.conn.cache_handle() + } + + pub async fn crdb(&self) -> Result { + self.conn.crdb().await + } + + pub async fn redis_cache(&self) -> Result { + self.conn.redis_cache().await + } + + pub async fn redis_cdn(&self) -> Result { + self.conn.redis_cdn().await + } + + pub async fn redis_job(&self) -> Result { + self.conn.redis_job().await + } + + pub async fn redis_mm(&self) -> Result { + self.conn.redis_mm().await + } + + pub async fn redis_user_presence(&self) -> Result { + self.conn.redis_user_presence().await + } + + pub async fn clickhouse(&self) -> GlobalResult { + self.conn.clickhouse().await + } + + // Backwards compatibility + pub fn op_ctx(&self) -> &rivet_operation::OperationContext<()> { + &self.op_ctx + } +} diff --git a/lib/chirp-workflow/core/src/ctx/test.rs b/lib/chirp-workflow/core/src/ctx/test.rs index 1c991d6ec..dc42556a5 100644 --- a/lib/chirp-workflow/core/src/ctx/test.rs +++ b/lib/chirp-workflow/core/src/ctx/test.rs @@ -7,6 +7,7 @@ use uuid::Uuid; use crate::{ ctx::{ message::{SubscriptionHandle, TailAnchor, TailAnchorResponse}, + workflow::SUB_WORKFLOW_RETRY, MessageCtx, OperationCtx, }, message::{Message, ReceivedMessage}, @@ -143,8 +144,7 @@ impl TestCtx { ) -> GlobalResult { tracing::info!(name=W::NAME, id=?workflow_id, "waiting for workflow"); - let period = Duration::from_millis(50); - let mut interval = tokio::time::interval(period); + let mut interval = tokio::time::interval(SUB_WORKFLOW_RETRY); loop { interval.tick().await; @@ -360,10 +360,6 @@ impl TestCtx { self.ts.saturating_sub(self.op_ctx.req_ts()) } - // pub fn perf(&self) -> &chirp_perf::PerfCtx { - // self.conn.perf() - // } - pub fn trace(&self) -> &[chirp_client::TraceEntry] { self.conn.trace() } diff --git a/lib/chirp-workflow/core/src/ctx/workflow.rs b/lib/chirp-workflow/core/src/ctx/workflow.rs index a85ae55f5..2959d2794 100644 --- a/lib/chirp-workflow/core/src/ctx/workflow.rs +++ b/lib/chirp-workflow/core/src/ctx/workflow.rs @@ -9,10 +9,11 @@ use crate::{ activity::ActivityId, ctx::{ActivityCtx, MessageCtx}, event::Event, + executable::{closure, AsyncResult, Executable}, message::Message, util::Location, - Activity, ActivityInput, DatabaseHandle, Executable, Listen, PulledWorkflow, RegistryHandle, - Signal, SignalRow, Workflow, WorkflowError, WorkflowInput, WorkflowResult, + Activity, ActivityInput, DatabaseHandle, Listen, PulledWorkflow, RegistryHandle, Signal, + SignalRow, Workflow, WorkflowError, WorkflowInput, WorkflowResult, }; // Time to delay a worker from retrying after an error @@ -22,7 +23,7 @@ const SIGNAL_RETRY: Duration = Duration::from_millis(100); // Most in-process signal poll tries const MAX_SIGNAL_RETRIES: usize = 16; // Poll interval when polling for a sub workflow in-process -const SUB_WORKFLOW_RETRY: Duration = Duration::from_millis(150); +pub const SUB_WORKFLOW_RETRY: Duration = Duration::from_millis(150); // Most in-process sub workflow poll tries const MAX_SUB_WORKFLOW_RETRIES: usize = 4; // Retry interval for failed db actions @@ -30,12 +31,13 @@ const DB_ACTION_RETRY: Duration = Duration::from_millis(150); // Most db action retries const MAX_DB_ACTION_RETRIES: usize = 5; -// TODO: Use generics to store input instead of a string +// TODO: Use generics to store input instead of a json value +// NOTE: Clonable because of inner arcs #[derive(Clone)] pub struct WorkflowCtx { - pub workflow_id: Uuid, + workflow_id: Uuid, /// Name of the workflow to run in the registry. - pub name: String, + name: String, create_ts: i64, ts: i64, ray_id: Uuid, @@ -577,7 +579,7 @@ impl WorkflowCtx { self.location_idx += 1; // Run workflow - let output = (workflow.run)(&mut ctx).await?; + let output = (workflow.run)(&mut ctx).await.map_err(GlobalError::raw)?; // TODO: RVT-3756 // Deserialize output @@ -667,6 +669,15 @@ impl WorkflowCtx { exec.execute(self).await } + /// Spawns a new thread to execute workflow steps in. + pub fn spawn(&mut self, f: F) -> tokio::task::JoinHandle> + where + F: for<'a> FnOnce(&'a mut WorkflowCtx) -> AsyncResult<'a, T> + Send + 'static, + { + let mut ctx = self.clone(); + tokio::task::spawn(async move { closure(f).execute(&mut ctx).await }) + } + /// Sends a signal. pub async fn signal( &mut self, @@ -699,7 +710,7 @@ impl WorkflowCtx { self.db .publish_signal_from_workflow( self.workflow_id, - self.full_location().as_ref(), + self.full_location().as_ref(), self.ray_id, workflow_id, signal_id, @@ -942,6 +953,14 @@ impl WorkflowCtx { } impl WorkflowCtx { + pub fn name(&self) -> &str { + &self.name + } + + pub fn workflow_id(&self) -> Uuid { + self.workflow_id + } + /// Timestamp at which this workflow run started. pub fn ts(&self) -> i64 { self.ts diff --git a/lib/chirp-workflow/core/src/db/postgres.rs b/lib/chirp-workflow/core/src/db/postgres.rs index 7fa70d3fc..e6882caf0 100644 --- a/lib/chirp-workflow/core/src/db/postgres.rs +++ b/lib/chirp-workflow/core/src/db/postgres.rs @@ -183,6 +183,7 @@ impl Database for DatabasePostgres { .map(|row| row.workflow_id) .collect::>(); + // TODO: Convert into union query // Fetch all events for all fetched workflows let ( activity_events, diff --git a/lib/chirp-workflow/core/src/executable.rs b/lib/chirp-workflow/core/src/executable.rs index e56e68785..dde5e56a1 100644 --- a/lib/chirp-workflow/core/src/executable.rs +++ b/lib/chirp-workflow/core/src/executable.rs @@ -14,9 +14,9 @@ pub trait Executable: Send { async fn execute(self, ctx: &mut WorkflowCtx) -> GlobalResult; } -type AsyncResult<'a, T> = Pin> + Send + 'a>>; +pub type AsyncResult<'a, T> = Pin> + Send + 'a>>; -// Closure executuable impl +// Closure executable impl #[async_trait] impl Executable for F where @@ -76,7 +76,7 @@ struct TupleHelper { // Must wrap all closured being used as executables in this function due to // https://github.com/rust-lang/rust/issues/70263 -pub fn closure(f: F) -> F +pub fn closure(f: F) -> F where F: for<'a> FnOnce(&'a mut WorkflowCtx) -> AsyncResult<'a, T> + Send, { diff --git a/lib/chirp-workflow/core/src/lib.rs b/lib/chirp-workflow/core/src/lib.rs index 9726e0724..e83964b4c 100644 --- a/lib/chirp-workflow/core/src/lib.rs +++ b/lib/chirp-workflow/core/src/lib.rs @@ -19,7 +19,6 @@ use activity::*; use ctx::*; use db::*; use error::*; -use executable::*; use operation::*; use registry::*; use signal::*; diff --git a/lib/chirp-workflow/core/src/signal.rs b/lib/chirp-workflow/core/src/signal.rs index d95dec106..f0e4c4cf2 100644 --- a/lib/chirp-workflow/core/src/signal.rs +++ b/lib/chirp-workflow/core/src/signal.rs @@ -44,14 +44,21 @@ pub trait Listen: Sized { /// ```` #[macro_export] macro_rules! join_signal { - (pub $join:ident, [$($signals:ident),*]) => { + (pub $join:ident, [$($signals:ident),* $(,)?]) => { pub enum $join { $($signals($signals)),* } join_signal!(@ $join, [$($signals),*]); }; - ($join:ident, [$($signals:ident),*]) => { + (pub($($vis:tt)*) $join:ident, [$($signals:ident),* $(,)?]) => { + pub($($vis)*) enum $join { + $($signals($signals)),* + } + + join_signal!(@ $join, [$($signals),*]); + }; + ($join:ident, [$($signals:ident),* $(,)?]) => { enum $join { $($signals($signals)),* } diff --git a/lib/chirp-workflow/macros/src/lib.rs b/lib/chirp-workflow/macros/src/lib.rs index 51500abe5..31a7bc139 100644 --- a/lib/chirp-workflow/macros/src/lib.rs +++ b/lib/chirp-workflow/macros/src/lib.rs @@ -284,8 +284,20 @@ pub fn signal(attr: TokenStream, item: TokenStream) -> TokenStream { let struct_ident = &item_struct.ident; + // If also a message, don't derive serde traits + let also_message = item_struct + .attrs + .iter() + .filter_map(|attr| attr.path().segments.last()) + .any(|seg| seg.ident == "message"); + let serde_derive = if also_message { + quote! {} + } else { + quote! { #[derive(serde::Serialize, serde::Deserialize)] } + }; + let expanded = quote! { - #[derive(serde::Serialize, serde::Deserialize)] + #serde_derive #item_struct impl Signal for #struct_ident { @@ -295,7 +307,7 @@ pub fn signal(attr: TokenStream, item: TokenStream) -> TokenStream { #[async_trait::async_trait] impl Listen for #struct_ident { async fn listen(ctx: &mut chirp_workflow::prelude::WorkflowCtx) -> chirp_workflow::prelude::WorkflowResult { - let row = ctx.listen_any(&[Self::NAME]).await?; + let row = ctx.listen_any(&[::NAME]).await?; Self::parse(&row.signal_name, row.body) } @@ -313,6 +325,18 @@ pub fn message(attr: TokenStream, item: TokenStream) -> TokenStream { let name = parse_macro_input!(attr as LitStr); let item_struct = parse_macro_input!(item as ItemStruct); + // If also a signal, don't derive serde traits + let also_signal = item_struct + .attrs + .iter() + .filter_map(|attr| attr.path().segments.last()) + .any(|seg| seg.ident == "signal"); + let serde_derive = if also_signal { + quote! {} + } else { + quote! { #[derive(serde::Serialize, serde::Deserialize)] } + }; + let config = match parse_msg_config(&item_struct.attrs) { Ok(x) => x, Err(err) => return err.into_compile_error().into(), @@ -322,25 +346,14 @@ pub fn message(attr: TokenStream, item: TokenStream) -> TokenStream { let tail_ttl = config.tail_ttl; let expanded = quote! { - #[derive(Debug, serde::Serialize, serde::Deserialize)] + #serde_derive + #[derive(Debug)] #item_struct impl Message for #struct_ident { const NAME: &'static str = #name; const TAIL_TTL: std::time::Duration = std::time::Duration::from_secs(#tail_ttl); } - - #[async_trait::async_trait] - impl Listen for #struct_ident { - async fn listen(ctx: &mut chirp_workflow::prelude::WorkflowCtx) -> chirp_workflow::prelude::WorkflowResult { - let row = ctx.listen_any(&[Self::NAME]).await?; - Self::parse(&row.signal_name, row.body) - } - - fn parse(_name: &str, body: serde_json::Value) -> chirp_workflow::prelude::WorkflowResult { - serde_json::from_value(body).map_err(WorkflowError::DeserializeActivityOutput) - } - } }; TokenStream::from(expanded) diff --git a/lib/convert/Cargo.toml b/lib/convert/Cargo.toml index 3901443ef..b0746b42a 100644 --- a/lib/convert/Cargo.toml +++ b/lib/convert/Cargo.toml @@ -15,6 +15,7 @@ types = { path = "../types/core" } util-mm = { package = "rivet-util-mm", path = "../../svc/pkg/mm/util" } cdn-namespace-get = { path = "../../svc/pkg/cdn/ops/namespace-get" } +cluster = { path = "../../svc/pkg/cluster" } game-get = { path = "../../svc/pkg/game/ops/get" } game-namespace-get = { path = "../../svc/pkg/game/ops/namespace-get" } game-namespace-list = { path = "../../svc/pkg/game/ops/namespace-list" } diff --git a/lib/convert/src/impls/admin.rs b/lib/convert/src/impls/admin.rs index fd911158d..f30db34bb 100644 --- a/lib/convert/src/impls/admin.rs +++ b/lib/convert/src/impls/admin.rs @@ -1,77 +1,78 @@ -use proto::backend::{self, pkg::*}; +// NOTE: This file should be moved to cluster/types.rs but there is a circular dependency with region-get + use rivet_api::models; use rivet_operation::prelude::*; use crate::{ApiFrom, ApiInto, ApiTryFrom}; -impl ApiFrom for backend::cluster::PoolType { - fn api_from(value: models::AdminClustersPoolType) -> backend::cluster::PoolType { +impl ApiFrom for cluster::types::PoolType { + fn api_from(value: models::AdminClustersPoolType) -> cluster::types::PoolType { match value { - models::AdminClustersPoolType::Job => backend::cluster::PoolType::Job, - models::AdminClustersPoolType::Gg => backend::cluster::PoolType::Gg, - models::AdminClustersPoolType::Ats => backend::cluster::PoolType::Ats, + models::AdminClustersPoolType::Job => cluster::types::PoolType::Job, + models::AdminClustersPoolType::Gg => cluster::types::PoolType::Gg, + models::AdminClustersPoolType::Ats => cluster::types::PoolType::Ats, } } } -impl ApiFrom for models::AdminClustersPoolType { - fn api_from(value: backend::cluster::PoolType) -> models::AdminClustersPoolType { +impl ApiFrom for models::AdminClustersPoolType { + fn api_from(value: cluster::types::PoolType) -> models::AdminClustersPoolType { match value { - backend::cluster::PoolType::Job => models::AdminClustersPoolType::Job, - backend::cluster::PoolType::Gg => models::AdminClustersPoolType::Gg, - backend::cluster::PoolType::Ats => models::AdminClustersPoolType::Ats, + cluster::types::PoolType::Job => models::AdminClustersPoolType::Job, + cluster::types::PoolType::Gg => models::AdminClustersPoolType::Gg, + cluster::types::PoolType::Ats => models::AdminClustersPoolType::Ats, } } } -impl ApiFrom for backend::cluster::Provider { - fn api_from(value: models::AdminClustersProvider) -> backend::cluster::Provider { +impl ApiFrom for cluster::types::Provider { + fn api_from(value: models::AdminClustersProvider) -> cluster::types::Provider { match value { - models::AdminClustersProvider::Linode => backend::cluster::Provider::Linode, + models::AdminClustersProvider::Linode => cluster::types::Provider::Linode, } } } -impl ApiFrom for models::AdminClustersProvider { - fn api_from(value: backend::cluster::Provider) -> models::AdminClustersProvider { +impl ApiFrom for models::AdminClustersProvider { + fn api_from(value: cluster::types::Provider) -> models::AdminClustersProvider { match value { - backend::cluster::Provider::Linode => models::AdminClustersProvider::Linode, + cluster::types::Provider::Linode => models::AdminClustersProvider::Linode, } } } -impl ApiFrom for backend::cluster::BuildDeliveryMethod { - fn api_from(value: models::AdminClustersBuildDeliveryMethod) -> backend::cluster::BuildDeliveryMethod { +impl ApiFrom for cluster::types::BuildDeliveryMethod { + fn api_from(value: models::AdminClustersBuildDeliveryMethod) -> cluster::types::BuildDeliveryMethod { match value { models::AdminClustersBuildDeliveryMethod::TrafficServer => { - backend::cluster::BuildDeliveryMethod::TrafficServer + cluster::types::BuildDeliveryMethod::TrafficServer } models::AdminClustersBuildDeliveryMethod::S3Direct => { - backend::cluster::BuildDeliveryMethod::S3Direct + cluster::types::BuildDeliveryMethod::S3Direct } } } } -impl ApiFrom for models::AdminClustersBuildDeliveryMethod { - fn api_from(value: backend::cluster::BuildDeliveryMethod) -> models::AdminClustersBuildDeliveryMethod { +impl ApiFrom for models::AdminClustersBuildDeliveryMethod { + fn api_from(value: cluster::types::BuildDeliveryMethod) -> models::AdminClustersBuildDeliveryMethod { match value { - backend::cluster::BuildDeliveryMethod::TrafficServer => { + cluster::types::BuildDeliveryMethod::TrafficServer => { models::AdminClustersBuildDeliveryMethod::TrafficServer } - backend::cluster::BuildDeliveryMethod::S3Direct => { + cluster::types::BuildDeliveryMethod::S3Direct => { models::AdminClustersBuildDeliveryMethod::S3Direct } } } } -impl ApiTryFrom for models::AdminClustersCluster { +impl ApiTryFrom for models::AdminClustersCluster { type Error = GlobalError; - fn api_try_from(value: backend::cluster::Cluster) -> GlobalResult { + fn api_try_from(value: cluster::types::Cluster) -> GlobalResult { Ok(models::AdminClustersCluster { - cluster_id: unwrap!(value.cluster_id).into(), + cluster_id: value.cluster_id, name_id: value.name_id, create_ts: value.create_ts, owner_team_id: value.owner_team_id.map(Into::into), @@ -79,17 +80,14 @@ impl ApiTryFrom for models::AdminClustersCluster { } } -impl ApiTryFrom for models::AdminClustersDatacenter { +impl ApiTryFrom for models::AdminClustersDatacenter { type Error = GlobalError; - fn api_try_from(value: backend::cluster::Datacenter) -> GlobalResult { + fn api_try_from(value: cluster::types::Datacenter) -> GlobalResult { Ok(models::AdminClustersDatacenter { - build_delivery_method: unwrap!(backend::cluster::BuildDeliveryMethod::from_i32( - value.build_delivery_method - )) - .api_into(), - cluster_id: unwrap!(value.cluster_id).into(), - datacenter_id: unwrap!(value.datacenter_id).into(), + build_delivery_method: value.build_delivery_method.api_into(), + cluster_id: value.cluster_id, + datacenter_id: value.datacenter_id, display_name: value.display_name, name_id: value.name_id, pools: value @@ -97,39 +95,38 @@ impl ApiTryFrom for models::AdminClustersDatacente .iter() .map(|p| { Ok(models::AdminClustersPool { - desired_count: unwrap!(p.desired_count.try_into()), - drain_timeout: unwrap!(p.drain_timeout.try_into()), + desired_count: p.desired_count.try_into()?, + drain_timeout: p.drain_timeout.try_into()?, hardware: p .hardware .iter() .map(|h| { - Ok(models::AdminClustersHardware { + models::AdminClustersHardware { provider_hardware: h.provider_hardware.clone(), - }) + } }) - .collect::, GlobalError>>()?, - min_count: unwrap!(p.min_count.try_into()), - max_count: unwrap!(p.max_count.try_into()), - pool_type: unwrap!(backend::cluster::PoolType::from_i32(p.pool_type)) - .api_into(), + .collect::>(), + min_count: p.min_count.try_into()?, + max_count: p.max_count.try_into()?, + pool_type: p.pool_type.clone().api_into(), }) }) - .collect::, GlobalError>>()?, - provider: unwrap!(backend::cluster::Provider::from_i32(value.provider)).api_into(), + .collect::>>()?, + provider: value.provider.api_into(), provider_api_token: value.provider_api_token, provider_datacenter_id: value.provider_datacenter_id, }) } } -impl ApiFrom for cluster::msg::datacenter_update::PoolUpdate { - fn api_from(value: models::AdminClustersPoolUpdate) -> cluster::msg::datacenter_update::PoolUpdate { - cluster::msg::datacenter_update::PoolUpdate { - pool_type: ApiInto::::api_into(value.pool_type) as i32, +impl ApiFrom for cluster::types::PoolUpdate { + fn api_from(value: models::AdminClustersPoolUpdate) -> cluster::types::PoolUpdate { + cluster::types::PoolUpdate { + pool_type: value.pool_type.api_into(), hardware: value .hardware .iter() - .map(|h| backend::cluster::Hardware { + .map(|h| cluster::types::Hardware { provider_hardware: h.provider_hardware.clone(), }) .collect(), @@ -139,4 +136,4 @@ impl ApiFrom for cluster::msg::datacenter_updat drain_timeout: value.drain_timeout.map(|d| d as u64), } } -} \ No newline at end of file +} diff --git a/proto/backend/cluster.proto b/proto/backend/cluster.proto index 8c901f555..a33b7b02c 100644 --- a/proto/backend/cluster.proto +++ b/proto/backend/cluster.proto @@ -1,36 +1,12 @@ -syntax = "proto3"; - -package rivet.backend.cluster; - -import "proto/common.proto"; -import "proto/backend/net.proto"; - -message Cluster { - rivet.common.Uuid cluster_id = 1; - string name_id = 2; - int64 create_ts = 3; - // Unset for the default cluster. - optional rivet.common.Uuid owner_team_id = 4; -} +// Kept for backwards compatibility (see svc/pkg/cluster/src/types.rs) -enum Provider { - LINODE = 0; -} -message Datacenter { - rivet.common.Uuid datacenter_id = 1; - rivet.common.Uuid cluster_id = 2; - string name_id = 3; - string display_name = 4; - int64 create_ts = 11; +syntax = "proto3"; - Provider provider = 5; - string provider_datacenter_id = 6; - optional string provider_api_token = 7; +package rivet.backend.cluster; - repeated Pool pools = 8; - BuildDeliveryMethod build_delivery_method = 9; - bool prebakes_enabled = 12; +message Pools { + repeated rivet.backend.cluster.Pool pools = 1; } message Pool { @@ -59,38 +35,3 @@ enum BuildDeliveryMethod { S3_DIRECT = 1; } -message Server { - rivet.common.Uuid server_id = 1; - rivet.common.Uuid cluster_id = 2; - rivet.common.Uuid datacenter_id = 3; - PoolType pool_type = 4; - optional string vlan_ip = 5; - optional string public_ip = 6; - - optional int64 cloud_destroy_ts = 7; - - // TODO: Add the rest of the sql columns -} - -enum TlsState { - CREATING = 0; - ACTIVE = 1; - RENEWING = 2; -} - -message ServerFilter { - bool filter_server_ids = 1; - repeated rivet.common.Uuid server_ids = 2; - - bool filter_cluster_ids = 3; - repeated rivet.common.Uuid cluster_ids = 4; - - bool filter_datacenter_ids = 5; - repeated rivet.common.Uuid datacenter_ids = 6; - - bool filter_pool_types = 7; - repeated PoolType pool_types = 8; - - bool filter_public_ips = 9; - repeated string public_ips = 10; -} diff --git a/sdks/full/go/admin/clusters/types.go b/sdks/full/go/admin/clusters/types.go index af67373ea..b8b0b1615 100644 --- a/sdks/full/go/admin/clusters/types.go +++ b/sdks/full/go/admin/clusters/types.go @@ -218,7 +218,7 @@ func (p Provider) Ptr() *Provider { type Server struct { ServerId uuid.UUID `json:"server_id"` - PublicIp string `json:"public_ip"` + PublicIp *string `json:"public_ip,omitempty"` _rawJSON json.RawMessage } diff --git a/sdks/full/openapi/openapi.yml b/sdks/full/openapi/openapi.yml index 90495ff22..d6cbdae07 100644 --- a/sdks/full/openapi/openapi.yml +++ b/sdks/full/openapi/openapi.yml @@ -10084,7 +10084,6 @@ components: type: string required: - server_id - - public_ip AdminClustersPoolUpdate: type: object properties: diff --git a/sdks/full/openapi_compat/openapi.yml b/sdks/full/openapi_compat/openapi.yml index 4534bbfa2..2a77e569c 100644 --- a/sdks/full/openapi_compat/openapi.yml +++ b/sdks/full/openapi_compat/openapi.yml @@ -206,7 +206,6 @@ components: type: string required: - server_id - - public_ip type: object AdminClustersUpdateDatacenterRequest: properties: diff --git a/sdks/full/rust-cli/docs/AdminClustersServer.md b/sdks/full/rust-cli/docs/AdminClustersServer.md index 2219721a9..08f7aa7f9 100644 --- a/sdks/full/rust-cli/docs/AdminClustersServer.md +++ b/sdks/full/rust-cli/docs/AdminClustersServer.md @@ -4,7 +4,7 @@ Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- -**public_ip** | **String** | | +**public_ip** | Option<**String**> | | [optional] **server_id** | [**uuid::Uuid**](uuid::Uuid.md) | | [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) diff --git a/sdks/full/rust-cli/src/models/admin_clusters_server.rs b/sdks/full/rust-cli/src/models/admin_clusters_server.rs index 81ac12b02..c1d6061f0 100644 --- a/sdks/full/rust-cli/src/models/admin_clusters_server.rs +++ b/sdks/full/rust-cli/src/models/admin_clusters_server.rs @@ -13,16 +13,16 @@ #[derive(Clone, Debug, PartialEq, Default, Serialize, Deserialize)] pub struct AdminClustersServer { - #[serde(rename = "public_ip")] - pub public_ip: String, + #[serde(rename = "public_ip", skip_serializing_if = "Option::is_none")] + pub public_ip: Option, #[serde(rename = "server_id")] pub server_id: uuid::Uuid, } impl AdminClustersServer { - pub fn new(public_ip: String, server_id: uuid::Uuid) -> AdminClustersServer { + pub fn new(server_id: uuid::Uuid) -> AdminClustersServer { AdminClustersServer { - public_ip, + public_ip: None, server_id, } } diff --git a/sdks/full/rust/docs/AdminClustersServer.md b/sdks/full/rust/docs/AdminClustersServer.md index 2219721a9..08f7aa7f9 100644 --- a/sdks/full/rust/docs/AdminClustersServer.md +++ b/sdks/full/rust/docs/AdminClustersServer.md @@ -4,7 +4,7 @@ Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- -**public_ip** | **String** | | +**public_ip** | Option<**String**> | | [optional] **server_id** | [**uuid::Uuid**](uuid::Uuid.md) | | [[Back to Model list]](../README.md#documentation-for-models) [[Back to API list]](../README.md#documentation-for-api-endpoints) [[Back to README]](../README.md) diff --git a/sdks/full/rust/src/models/admin_clusters_server.rs b/sdks/full/rust/src/models/admin_clusters_server.rs index 81ac12b02..c1d6061f0 100644 --- a/sdks/full/rust/src/models/admin_clusters_server.rs +++ b/sdks/full/rust/src/models/admin_clusters_server.rs @@ -13,16 +13,16 @@ #[derive(Clone, Debug, PartialEq, Default, Serialize, Deserialize)] pub struct AdminClustersServer { - #[serde(rename = "public_ip")] - pub public_ip: String, + #[serde(rename = "public_ip", skip_serializing_if = "Option::is_none")] + pub public_ip: Option, #[serde(rename = "server_id")] pub server_id: uuid::Uuid, } impl AdminClustersServer { - pub fn new(public_ip: String, server_id: uuid::Uuid) -> AdminClustersServer { + pub fn new(server_id: uuid::Uuid) -> AdminClustersServer { AdminClustersServer { - public_ip, + public_ip: None, server_id, } } diff --git a/sdks/full/typescript/archive.tgz b/sdks/full/typescript/archive.tgz index 9206d47b9..dad954eca 100644 --- a/sdks/full/typescript/archive.tgz +++ b/sdks/full/typescript/archive.tgz @@ -1,3 +1,3 @@ version https://git-lfs.github.com/spec/v1 -oid sha256:6dd57b0829c24ebac5513e6bec6c92519111cb03619a811f4150d151fb440078 -size 641144 +oid sha256:88b586c4470a0f6097bf139b6ea9cc4330da3f27e7e8b715291e48c34da5edb1 +size 641236 diff --git a/sdks/full/typescript/src/core/fetcher/Fetcher.ts b/sdks/full/typescript/src/core/fetcher/Fetcher.ts index cb5ee9c1d..19de5d475 100644 --- a/sdks/full/typescript/src/core/fetcher/Fetcher.ts +++ b/sdks/full/typescript/src/core/fetcher/Fetcher.ts @@ -73,13 +73,15 @@ async function fetcherImpl(args: Fetcher.Args): Promise => { const controller = new AbortController(); let abortId = undefined; if (args.timeoutMs != null) { abortId = setTimeout(() => controller.abort(), args.timeoutMs); } - const response = await fetch(url, { + const response = await fetchFn(url, { method: args.method, headers, body, diff --git a/sdks/full/typescript/types/api/resources/admin/resources/clusters/resources/common/types/Server.d.ts b/sdks/full/typescript/types/api/resources/admin/resources/clusters/resources/common/types/Server.d.ts index 142110bc5..6f69f68d4 100644 --- a/sdks/full/typescript/types/api/resources/admin/resources/clusters/resources/common/types/Server.d.ts +++ b/sdks/full/typescript/types/api/resources/admin/resources/clusters/resources/common/types/Server.d.ts @@ -3,5 +3,5 @@ */ export interface Server { serverId: string; - publicIp: string; + publicIp?: string; } diff --git a/sdks/full/typescript/types/serialization/resources/admin/resources/clusters/resources/common/types/Server.d.ts b/sdks/full/typescript/types/serialization/resources/admin/resources/clusters/resources/common/types/Server.d.ts index ec3a18be9..ce1e71296 100644 --- a/sdks/full/typescript/types/serialization/resources/admin/resources/clusters/resources/common/types/Server.d.ts +++ b/sdks/full/typescript/types/serialization/resources/admin/resources/clusters/resources/common/types/Server.d.ts @@ -8,6 +8,6 @@ export declare const Server: core.serialization.ObjectSchema(args: Fetcher.Args): Promise => { const controller = new AbortController(); let abortId = undefined; if (args.timeoutMs != null) { abortId = setTimeout(() => controller.abort(), args.timeoutMs); } - - // We assume here that the environment is providing fetch functionality - const response = await fetch(url, { + const response = await fetchFn(url, { method: args.method, headers, body, diff --git a/svc/Cargo.lock b/svc/Cargo.lock index 275cac882..91e8c490a 100644 --- a/svc/Cargo.lock +++ b/svc/Cargo.lock @@ -103,14 +103,7 @@ dependencies = [ "chirp-client", "chirp-workflow", "chrono", - "cluster-datacenter-get", - "cluster-datacenter-list", - "cluster-datacenter-resolve-for-name-id", - "cluster-get", - "cluster-list", - "cluster-server-destroy-with-filter", - "cluster-server-get", - "cluster-server-list", + "cluster", "http 0.2.12", "hyper", "lazy_static", @@ -125,7 +118,6 @@ dependencies = [ "rivet-matchmaker", "rivet-operation", "rivet-pools", - "rivet-util-cluster", "rivet-util-mm", "s3-util", "serde", @@ -253,7 +245,7 @@ dependencies = [ "cloud-namespace-token-public-create", "cloud-version-get", "cloud-version-publish", - "cluster-datacenter-list", + "cluster", "custom-user-avatar-list-for-game", "custom-user-avatar-upload-complete", "faker-region", @@ -307,7 +299,6 @@ dependencies = [ "rivet-health-checks", "rivet-operation", "rivet-pools", - "rivet-util-cluster", "rivet-util-job", "rivet-util-mm", "rivet-util-nsfw", @@ -777,10 +768,7 @@ dependencies = [ "async-trait", "chirp-client", "chrono", - "cluster-datacenter-get", - "cluster-datacenter-tls-get", - "cluster-server-get", - "cluster-server-resolve-for-ip", + "cluster", "http 0.2.12", "hyper", "lazy_static", @@ -791,7 +779,6 @@ dependencies = [ "rivet-health-checks", "rivet-operation", "rivet-pools", - "rivet-util-cluster", "serde", "serde_json", "thiserror", @@ -859,7 +846,7 @@ dependencies = [ "cdn-namespace-domain-create", "chirp-client", "chrono", - "cluster-server-list", + "cluster", "faker-cdn-site", "faker-game", "faker-game-namespace", @@ -2400,58 +2387,35 @@ dependencies = [ ] [[package]] -name = "cluster-datacenter-get" +name = "cluster" version = "0.0.1" dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-datacenter-list" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-datacenter-location-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", + "acme-lib", + "anyhow", + "chirp-workflow", + "cloudflare", + "hex", + "http 0.2.12", + "include_dir", + "indoc 1.0.9", "ip-info", + "lazy_static", + "linode", + "merkle_hash", + "nomad-util", + "nomad_client", + "rand", + "rivet-metrics", "rivet-operation", + "rivet-runtime", + "s3-util", + "serde", "sqlx", -] - -[[package]] -name = "cluster-datacenter-resolve-for-name-id" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-datacenter-tls-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", + "ssh2", + "thiserror", + "token-create", + "tokio", + "trust-dns-resolver", ] [[package]] @@ -2459,12 +2423,11 @@ name = "cluster-datacenter-tls-renew" version = "0.0.1" dependencies = [ "chirp-client", - "chirp-worker", - "cluster-datacenter-get", + "chirp-workflow", + "cluster", "rivet-connection", "rivet-health-checks", "rivet-metrics", - "rivet-operation", "rivet-runtime", "sqlx", "tokio", @@ -2472,35 +2435,17 @@ dependencies = [ "tracing-subscriber", ] -[[package]] -name = "cluster-datacenter-topology-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "lazy_static", - "nomad-util", - "nomad_client", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - [[package]] name = "cluster-default-update" version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-get", - "cluster-datacenter-list", - "cluster-get", - "prost 0.10.4", + "chirp-workflow", + "cluster", "reqwest", "rivet-connection", - "rivet-operation", "rivet-pools", - "rivet-util-cluster", "serde", "serde_json", "tokio", @@ -2509,212 +2454,40 @@ dependencies = [ "uuid", ] -[[package]] -name = "cluster-fix-tls" -version = "0.0.1" -dependencies = [ - "acme-lib", - "anyhow", - "chirp-client", - "chirp-worker", - "chrono", - "cloudflare", - "cluster-datacenter-get", - "cluster-datacenter-list", - "cluster-datacenter-topology-get", - "http 0.2.12", - "include_dir", - "indoc 1.0.9", - "lazy_static", - "linode-instance-type-get", - "linode-server-destroy", - "linode-server-provision", - "maplit", - "nomad-util", - "openssl", - "rivet-connection", - "rivet-convert", - "rivet-health-checks", - "rivet-metrics", - "rivet-operation", - "rivet-runtime", - "rivet-util-cluster", - "s3-util", - "serde_yaml", - "ssh2", - "thiserror", - "token-create", - "tokio", - "tracing", - "tracing-subscriber", - "trust-dns-resolver", -] - [[package]] name = "cluster-gc" version = "0.0.1" dependencies = [ "chirp-client", - "chirp-worker", - "cluster-datacenter-get", + "chirp-workflow", + "cluster", "rivet-connection", "rivet-health-checks", "rivet-metrics", - "rivet-operation", "rivet-runtime", - "rivet-util-cluster", "sqlx", "tokio", "tracing", "tracing-subscriber", ] -[[package]] -name = "cluster-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-get-for-game" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "rivet-util-cluster", - "sqlx", -] - -[[package]] -name = "cluster-list" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - [[package]] name = "cluster-metrics-publish" version = "0.0.1" dependencies = [ "chirp-client", - "chirp-worker", - "cluster-datacenter-get", + "chirp-workflow", + "cluster", "rivet-connection", "rivet-health-checks", "rivet-metrics", - "rivet-operation", "rivet-runtime", - "rivet-util-cluster", "sqlx", "tokio", "tracing", "tracing-subscriber", ] -[[package]] -name = "cluster-resolve-for-name-id" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-server-destroy-with-filter" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "cluster-server-list", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-server-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-server-list" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-server-resolve-for-ip" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "prost 0.10.4", - "rivet-operation", - "sqlx", -] - -[[package]] -name = "cluster-worker" -version = "0.0.1" -dependencies = [ - "acme-lib", - "anyhow", - "chirp-client", - "chirp-worker", - "chrono", - "cloudflare", - "cluster-datacenter-get", - "cluster-datacenter-list", - "cluster-datacenter-topology-get", - "http 0.2.12", - "include_dir", - "indoc 1.0.9", - "lazy_static", - "linode-instance-type-get", - "linode-server-destroy", - "linode-server-provision", - "maplit", - "nomad-util", - "nomad_client", - "openssl", - "rivet-convert", - "rivet-health-checks", - "rivet-metrics", - "rivet-runtime", - "rivet-util-cluster", - "s3-util", - "serde_yaml", - "sqlx", - "ssh2", - "thiserror", - "token-create", - "trust-dns-resolver", -] - [[package]] name = "combine" version = "4.6.6" @@ -3732,6 +3505,8 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", + "chirp-workflow", + "cluster", "faker-team", "game-validate", "prost 0.10.4", @@ -5102,84 +4877,40 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0717cef1bc8b636c6e1c1bbdefc09e6322da8a9321966e8928ef80d20f7f770f" [[package]] -name = "linode-gc" +name = "linode" version = "0.0.1" dependencies = [ - "chirp-client", - "chirp-worker", + "chirp-workflow", "chrono", + "cluster", + "rand", "reqwest", - "rivet-connection", - "rivet-health-checks", - "rivet-metrics", - "rivet-operation", - "rivet-runtime", - "rivet-util-cluster", - "rivet-util-linode", "serde", "serde_json", "sqlx", - "tokio", - "tracing", - "tracing-subscriber", -] - -[[package]] -name = "linode-instance-type-get" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "rivet-operation", - "rivet-util-cluster", - "rivet-util-linode", - "sqlx", -] - -[[package]] -name = "linode-server-destroy" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "cluster-datacenter-get", - "linode-server-provision", - "reqwest", - "rivet-operation", - "rivet-util-cluster", - "rivet-util-linode", - "sqlx", + "ssh-key", ] [[package]] -name = "linode-server-provision" +name = "linode-gc" version = "0.0.1" dependencies = [ "chirp-client", - "chirp-worker", - "cluster-datacenter-get", - "linode-server-destroy", + "chirp-workflow", + "chrono", + "cluster", + "linode", "reqwest", - "rivet-operation", - "rivet-util-cluster", - "rivet-util-linode", - "sqlx", -] - -[[package]] -name = "linode-worker" -version = "0.0.1" -dependencies = [ - "chirp-client", - "chirp-worker", - "cluster-datacenter-get", - "rivet-convert", + "rivet-connection", "rivet-health-checks", "rivet-metrics", "rivet-runtime", - "rivet-util-cluster", - "rivet-util-linode", + "serde", + "serde_json", "sqlx", + "tokio", + "tracing", + "tracing-subscriber", ] [[package]] @@ -5584,6 +5315,7 @@ dependencies = [ "build-get", "chirp-client", "chirp-worker", + "cluster", "faker-build", "faker-game", "faker-region", @@ -5853,6 +5585,7 @@ dependencies = [ "chirp-client", "chirp-worker", "chrono", + "cluster", "csv", "faker-build", "faker-game", @@ -5915,13 +5648,11 @@ dependencies = [ "cf-custom-hostname-worker", "chirp-client", "cloud-worker", - "cluster-worker", "external-worker", "game-user-worker", "job-log-worker", "job-run-worker", "kv-worker", - "linode-worker", "mm-worker", "rivet-connection", "rivet-health-checks", @@ -5946,6 +5677,8 @@ name = "monolith-workflow-worker" version = "0.0.1" dependencies = [ "chirp-workflow", + "cluster", + "linode", "rivet-health-checks", "rivet-metrics", "rivet-runtime", @@ -6026,7 +5759,9 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", + "chirp-workflow", "chrono", + "cluster", "futures-util", "indoc 1.0.9", "lazy_static", @@ -6938,8 +6673,8 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-get", - "cluster-datacenter-location-get", + "chirp-workflow", + "cluster", "faker-region", "prost 0.10.4", "rivet-operation", @@ -6952,11 +6687,11 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-list", + "chirp-workflow", + "cluster", "faker-region", "prost 0.10.4", "rivet-operation", - "rivet-util-cluster", "sqlx", ] @@ -6966,8 +6701,8 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-list", - "cluster-get-for-game", + "chirp-workflow", + "cluster", "faker-region", "prost 0.10.4", "rivet-operation", @@ -6995,7 +6730,8 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-get", + "chirp-workflow", + "cluster", "faker-region", "prost 0.10.4", "region-get", @@ -7010,7 +6746,8 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-get", + "chirp-workflow", + "cluster", "faker-region", "prost 0.10.4", "region-get", @@ -7240,6 +6977,7 @@ dependencies = [ "cdn-namespace-get", "chirp-client", "chrono", + "cluster", "game-get", "game-namespace-get", "game-namespace-list", @@ -7567,20 +7305,6 @@ dependencies = [ name = "rivet-util-cdn" version = "0.1.0" -[[package]] -name = "rivet-util-cluster" -version = "0.1.0" -dependencies = [ - "hex", - "lazy_static", - "merkle_hash", - "rivet-metrics", - "rivet-util", - "tokio", - "types", - "uuid", -] - [[package]] name = "rivet-util-env" version = "0.1.0" @@ -7612,19 +7336,6 @@ dependencies = [ name = "rivet-util-kv" version = "0.1.0" -[[package]] -name = "rivet-util-linode" -version = "0.1.0" -dependencies = [ - "chrono", - "rand", - "reqwest", - "rivet-operation", - "serde", - "serde_json", - "ssh-key", -] - [[package]] name = "rivet-util-macros" version = "0.1.0" @@ -8091,19 +7802,6 @@ dependencies = [ "syn 2.0.58", ] -[[package]] -name = "serde_yaml" -version = "0.9.34+deprecated" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" -dependencies = [ - "indexmap 2.2.6", - "itoa 1.0.11", - "ryu", - "serde", - "unsafe-libyaml", -] - [[package]] name = "sha1" version = "0.10.6" @@ -8934,12 +8632,11 @@ version = "0.0.1" dependencies = [ "chirp-client", "chirp-worker", - "cluster-datacenter-get", - "cluster-datacenter-list", - "linode-instance-type-get", + "chirp-workflow", + "cluster", + "linode", "prost 0.10.4", "rivet-operation", - "rivet-util-cluster", ] [[package]] @@ -9547,12 +9244,6 @@ version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39ec24b3121d976906ece63c9daad25b85969647682eee313cb5779fdd69e14e" -[[package]] -name = "unsafe-libyaml" -version = "0.2.11" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" - [[package]] name = "untrusted" version = "0.7.1" diff --git a/svc/Cargo.toml b/svc/Cargo.toml index 44e05006b..0e62def32 100644 --- a/svc/Cargo.toml +++ b/svc/Cargo.toml @@ -59,26 +59,11 @@ members = [ "pkg/cloud/ops/version-get", "pkg/cloud/ops/version-publish", "pkg/cloud/worker", - "pkg/cluster/ops/datacenter-get", - "pkg/cluster/ops/datacenter-list", - "pkg/cluster/ops/datacenter-location-get", - "pkg/cluster/ops/datacenter-resolve-for-name-id", - "pkg/cluster/ops/datacenter-tls-get", - "pkg/cluster/ops/datacenter-topology-get", - "pkg/cluster/ops/get", - "pkg/cluster/ops/get-for-game", - "pkg/cluster/ops/list", - "pkg/cluster/ops/resolve-for-name-id", - "pkg/cluster/ops/server-destroy-with-filter", - "pkg/cluster/ops/server-get", - "pkg/cluster/ops/server-list", - "pkg/cluster/ops/server-resolve-for-ip", + "pkg/cluster", "pkg/cluster/standalone/datacenter-tls-renew", "pkg/cluster/standalone/default-update", - "pkg/cluster/standalone/fix-tls", "pkg/cluster/standalone/gc", "pkg/cluster/standalone/metrics-publish", - "pkg/cluster/worker", "pkg/custom-user-avatar/ops/list-for-game", "pkg/custom-user-avatar/ops/upload-complete", "pkg/debug/ops/email-res", @@ -151,11 +136,8 @@ members = [ "pkg/kv/ops/get", "pkg/kv/ops/list", "pkg/kv/worker", - "pkg/linode/ops/instance-type-get", - "pkg/linode/ops/server-destroy", - "pkg/linode/ops/server-provision", + "pkg/linode", "pkg/linode/standalone/gc", - "pkg/linode/worker", "pkg/load-test/standalone/api-cloud", "pkg/load-test/standalone/mm", "pkg/load-test/standalone/mm-sustain", diff --git a/svc/api/admin/Cargo.toml b/svc/api/admin/Cargo.toml index ced0d77b6..697fa1c95 100644 --- a/svc/api/admin/Cargo.toml +++ b/svc/api/admin/Cargo.toml @@ -39,17 +39,9 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ ] } url = "2.2.2" uuid = { version = "1", features = ["v4"] } -util-cluster = { package = "rivet-util-cluster", path = "../../pkg/cluster/util" } util-mm = { package = "rivet-util-mm", path = "../../pkg/mm/util" } -cluster-get = { path = "../../pkg/cluster/ops/get" } -cluster-list = { path = "../../pkg/cluster/ops/list" } -cluster-server-get = { path = "../../pkg/cluster/ops/server-get" } -cluster-server-destroy-with-filter = { path = "../../pkg/cluster/ops/server-destroy-with-filter" } -cluster-server-list = { path = "../../pkg/cluster/ops/server-list" } -cluster-datacenter-list = { path = "../../pkg/cluster/ops/datacenter-list" } -cluster-datacenter-get = { path = "../../pkg/cluster/ops/datacenter-get" } -cluster-datacenter-resolve-for-name-id = { path = "../../pkg/cluster/ops/datacenter-resolve-for-name-id" } +cluster = { path = "../../pkg/cluster" } token-create = { path = "../../pkg/token/ops/create" } [dev-dependencies] diff --git a/svc/api/admin/src/route/clusters/datacenters.rs b/svc/api/admin/src/route/clusters/datacenters.rs index 001fa0a27..f3f9dfb3d 100644 --- a/svc/api/admin/src/route/clusters/datacenters.rs +++ b/svc/api/admin/src/route/clusters/datacenters.rs @@ -1,8 +1,7 @@ use api_helper::{anchor::WatchIndexQuery, ctx::Ctx}; -use proto::backend; use rivet_api::models; -use rivet_convert::{ApiFrom, ApiInto, ApiTryFrom}; -use rivet_operation::prelude::{proto::backend::pkg::cluster, *}; +use rivet_convert::{ApiInto, ApiTryInto}; +use rivet_operation::prelude::*; use serde_json::{json, Value}; use crate::auth::Auth; @@ -13,21 +12,23 @@ pub async fn list( cluster_id: Uuid, _watch_index: WatchIndexQuery, ) -> GlobalResult { - let response = op!([ctx] cluster_datacenter_list { - cluster_ids: vec![cluster_id.into()], - }) - .await?; - - let datacenter_ids = unwrap!(response.clusters.first()).datacenter_ids.clone(); - - let datacenters = op!([ctx] cluster_datacenter_get { - datacenter_ids: datacenter_ids.into_iter().map(Into::into).collect() - }) - .await? - .datacenters - .into_iter() - .map(models::AdminClustersDatacenter::api_try_from) - .collect::>>()?; + let datacenters_res = ctx + .op(cluster::ops::datacenter::list::Input { + cluster_ids: vec![cluster_id], + }) + .await?; + + let datacenter_ids = unwrap!(datacenters_res.clusters.first()) + .datacenter_ids + .clone(); + + let datacenters = ctx + .op(cluster::ops::datacenter::get::Input { datacenter_ids }) + .await? + .datacenters + .into_iter() + .map(ApiTryInto::::api_try_into) + .collect::>>()?; Ok(models::AdminClustersListDatacentersResponse { datacenters }) } @@ -39,11 +40,12 @@ pub async fn create( body: models::AdminClustersCreateDatacenterRequest, ) -> GlobalResult { // Make sure the cluster exists - let clusters = op!([ctx] cluster_get { - cluster_ids: vec![cluster_id.into()], - }) - .await? - .clusters; + let clusters = ctx + .op(cluster::ops::get::Input { + cluster_ids: vec![cluster_id], + }) + .await? + .clusters; if clusters.is_empty() { bail_with!(CLUSTER_NOT_FOUND); @@ -55,9 +57,9 @@ pub async fn create( // When creating a datacenter, an empty pool of each type is added. This // is to make sure that the datacenter starts in a valid state. let pools = vec![ - backend::cluster::Pool { - pool_type: backend::cluster::PoolType::Job as i32, - hardware: vec![backend::cluster::Hardware { + cluster::types::Pool { + pool_type: cluster::types::PoolType::Job, + hardware: vec![cluster::types::Hardware { provider_hardware: "g6-nanode-1".to_string(), }], desired_count: 0, @@ -65,9 +67,9 @@ pub async fn create( max_count: 0, drain_timeout, }, - backend::cluster::Pool { - pool_type: backend::cluster::PoolType::Gg as i32, - hardware: vec![backend::cluster::Hardware { + cluster::types::Pool { + pool_type: cluster::types::PoolType::Gg, + hardware: vec![cluster::types::Hardware { provider_hardware: "g6-nanode-1".to_string(), }], desired_count: 0, @@ -75,9 +77,9 @@ pub async fn create( max_count: 0, drain_timeout, }, - backend::cluster::Pool { - pool_type: backend::cluster::PoolType::Ats as i32, - hardware: vec![backend::cluster::Hardware { + cluster::types::Pool { + pool_type: cluster::types::PoolType::Ats, + hardware: vec![cluster::types::Hardware { provider_hardware: "g6-nanode-1".to_string(), }], desired_count: 0, @@ -87,23 +89,35 @@ pub async fn create( }, ]; - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: body.name_id.clone(), - display_name: body.display_name.clone(), - - provider: backend::cluster::Provider::api_from(body.provider) as i32, - provider_datacenter_id: body.provider_datacenter_id.clone(), - provider_api_token: None, - - pools: pools, - - build_delivery_method: backend::cluster::BuildDeliveryMethod::api_from(body.build_delivery_method) as i32, - prebakes_enabled: body.prebakes_enabled, - }) + let mut sub = ctx + .subscribe::(&json!({ + "datacenter_id": datacenter_id, + })) + .await?; + + ctx.tagged_signal( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id, + name_id: body.name_id.clone(), + display_name: body.display_name.clone(), + + provider: body.provider.api_into(), + provider_datacenter_id: body.provider_datacenter_id.clone(), + provider_api_token: None, + + pools, + + build_delivery_method: body.build_delivery_method.api_into(), + prebakes_enabled: body.prebakes_enabled, + }, + ) .await?; + sub.next().await?; + Ok(models::AdminClustersCreateDatacenterResponse { datacenter_id }) } @@ -114,33 +128,35 @@ pub async fn update( datacenter_id: Uuid, body: models::AdminClustersUpdateDatacenterRequest, ) -> GlobalResult { + let datacenters = ctx + .op(cluster::ops::datacenter::get::Input { + datacenter_ids: vec![datacenter_id], + }) + .await? + .datacenters; + + let datacenter = unwrap_with!(datacenters.first(), CLUSTER_DATACENTER_NOT_FOUND); + // Make sure that the datacenter is part of the cluster - let datacenters = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await? - .datacenters; - - let datacenter = match datacenters.first() { - Some(d) => d, - None => bail_with!(CLUSTER_DATACENTER_NOT_FOUND), - }; - - if datacenter.cluster_id != Some(cluster_id.into()) { + if datacenter.cluster_id != cluster_id { bail_with!(CLUSTER_DATACENTER_NOT_IN_CLUSTER); } - msg!([ctx] cluster::msg::datacenter_update(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - pools: body.pools - .iter() - .cloned() - .map(ApiInto::api_into) - .collect::>(), - prebakes_enabled: body.prebakes_enabled, - }) - .await - .unwrap(); + ctx.tagged_signal( + &json!({ + "datacenter_id": datacenter_id, + }), + cluster::workflows::datacenter::Update { + pools: body + .pools + .iter() + .cloned() + .map(ApiInto::api_into) + .collect::>(), + prebakes_enabled: body.prebakes_enabled, + }, + ) + .await?; Ok(json!({})) } diff --git a/svc/api/admin/src/route/clusters/mod.rs b/svc/api/admin/src/route/clusters/mod.rs index 05c42c64d..87560a7df 100644 --- a/svc/api/admin/src/route/clusters/mod.rs +++ b/svc/api/admin/src/route/clusters/mod.rs @@ -1,7 +1,8 @@ use api_helper::{anchor::WatchIndexQuery, ctx::Ctx}; use rivet_api::models; -use rivet_convert::ApiTryFrom; -use rivet_operation::prelude::{proto::backend::pkg::cluster, *}; +use rivet_convert::ApiTryInto; +use rivet_operation::prelude::*; +use serde_json::json; use crate::auth::Auth; @@ -13,16 +14,15 @@ pub async fn list( ctx: Ctx, _watch_index: WatchIndexQuery, ) -> GlobalResult { - let cluster_ids = op!([ctx] cluster_list {}).await?.cluster_ids; + let cluster_ids = ctx.op(cluster::ops::list::Input {}).await?.cluster_ids; - let clusters = op!([ctx] cluster_get { - cluster_ids: cluster_ids.into_iter().map(Into::into).collect() - }) - .await? - .clusters - .into_iter() - .map(models::AdminClustersCluster::api_try_from) - .collect::>>()?; + let clusters = ctx + .op(cluster::ops::get::Input { cluster_ids }) + .await? + .clusters + .into_iter() + .map(ApiTryInto::::api_try_into) + .collect::>>()?; Ok(models::AdminClustersListClustersResponse { clusters }) } @@ -34,12 +34,24 @@ pub async fn create( ) -> GlobalResult { let cluster_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - owner_team_id: body.owner_team_id.map(Into::into), - name_id: body.name_id, - }) + let tags = json!({ + "cluster_id": cluster_id, + }); + let mut sub = ctx + .subscribe::(&tags) + .await?; + + ctx.dispatch_tagged_workflow( + &tags, + cluster::workflows::cluster::Input { + cluster_id, + owner_team_id: body.owner_team_id, + name_id: body.name_id, + }, + ) .await?; + sub.next().await?; + Ok(models::AdminClustersCreateClusterResponse { cluster_id }) } diff --git a/svc/api/admin/src/route/clusters/servers.rs b/svc/api/admin/src/route/clusters/servers.rs index f0a8110f2..78ca0d3b7 100644 --- a/svc/api/admin/src/route/clusters/servers.rs +++ b/svc/api/admin/src/route/clusters/servers.rs @@ -1,10 +1,9 @@ +use std::{net::Ipv4Addr, str::FromStr}; + use api_helper::{anchor::WatchIndexQuery, ctx::Ctx}; use rivet_api::models; -use rivet_convert::ApiFrom; -use rivet_operation::prelude::{ - proto::backend::{self, pkg::*}, - *, -}; +use rivet_convert::ApiInto; +use rivet_operation::prelude::*; use serde::Deserialize; use serde_json::{json, Value}; @@ -19,48 +18,37 @@ pub struct ServerFilterQuery { } impl ServerFilterQuery { - async fn convert_to_proto( + async fn convert( &self, ctx: &Ctx, cluster_id: Uuid, - ) -> GlobalResult { - let mut filter = backend::cluster::ServerFilter::default(); - - filter.filter_cluster_ids = true; - filter.cluster_ids = vec![cluster_id.into()]; - - if let Some(server_id) = self.server_id { - filter.filter_server_ids = true; - filter.server_ids = vec![server_id.into()]; - } - - if let Some(name_id) = &self.datacenter { - // Look up datacenter - let resolve_res = op!([ctx] cluster_datacenter_resolve_for_name_id { - cluster_id: Some(cluster_id.into()), - name_ids: vec![name_id.clone()], - }) - .await?; - let datacenter = unwrap!(resolve_res.datacenters.first(), "datacenter not found"); - - // Filter datacenters - filter.filter_datacenter_ids = true; - filter.datacenter_ids = vec![unwrap!(datacenter.datacenter_id)]; - } - - if let Some(pool) = self.pool { - let pool_type = >::api_from(pool); - - filter.filter_pool_types = true; - filter.pool_types = vec![pool_type as i32]; - } - - if let Some(public_ip) = &self.public_ip { - filter.filter_public_ips = true; - filter.public_ips = vec![public_ip.clone()]; - } - - Ok(filter) + ) -> GlobalResult { + Ok(cluster::types::Filter { + cluster_ids: Some(vec![cluster_id]), + server_ids: self.server_id.map(|x| vec![x]), + datacenter_ids: if let Some(name_id) = &self.datacenter { + // Look up datacenter + let resolve_res = ctx + .op(cluster::ops::datacenter::resolve_for_name_id::Input { + cluster_id, + name_ids: vec![name_id.clone()], + }) + .await?; + let datacenter = unwrap!(resolve_res.datacenters.first(), "datacenter not found"); + + // Filter datacenters + Some(vec![datacenter.datacenter_id]) + } else { + None + }, + pool_types: self.pool.map(ApiInto::api_into).map(|x| vec![x]), + public_ips: self + .public_ip + .as_deref() + .map(Ipv4Addr::from_str) + .transpose()? + .map(|x| vec![x]), + }) } } @@ -71,20 +59,20 @@ pub async fn list( _watch_index: WatchIndexQuery, query: ServerFilterQuery, ) -> GlobalResult { - let filter = query.convert_to_proto(&ctx, cluster_id).await?; - - let servers_res = op!([ctx] cluster_server_list { - filter: Some(filter) - }) - .await?; + let servers_res = ctx + .op(cluster::ops::server::list::Input { + filter: query.convert(&ctx, cluster_id).await?, + include_destroyed: false, + }) + .await?; let servers = servers_res .servers .iter() - .map(|x| { + .map(|server| { GlobalResult::Ok(models::AdminClustersServer { - server_id: unwrap!(x.server_id).as_uuid(), - public_ip: x.public_ip.clone().unwrap_or_default(), + server_id: server.server_id, + public_ip: server.public_ip.map(|ip| ip.to_string()), }) }) .collect::>>()?; @@ -99,11 +87,8 @@ pub async fn taint( _body: serde_json::Value, query: ServerFilterQuery, ) -> GlobalResult { - let filter = query.convert_to_proto(&ctx, cluster_id).await?; - - let request_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::server_taint(request_id) { - filter: Some(filter), + ctx.op(cluster::ops::server::taint_with_filter::Input { + filter: query.convert(&ctx, cluster_id).await?, }) .await?; @@ -117,10 +102,8 @@ pub async fn destroy( _body: serde_json::Value, query: ServerFilterQuery, ) -> GlobalResult { - let filter = query.convert_to_proto(&ctx, cluster_id).await?; - - op!([ctx] cluster_server_destroy_with_filter { - filter: Some(filter) + ctx.op(cluster::ops::server::destroy_with_filter::Input { + filter: query.convert(&ctx, cluster_id).await?, }) .await?; diff --git a/svc/api/cloud/Cargo.toml b/svc/api/cloud/Cargo.toml index 5aad793ab..b7f8dc162 100644 --- a/svc/api/cloud/Cargo.toml +++ b/svc/api/cloud/Cargo.toml @@ -34,7 +34,6 @@ tokio = { version = "1.29" } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } url = "2.2.2" -util-cluster = { package = "rivet-util-cluster", path = "../../pkg/cluster/util" } util-job = { package = "rivet-util-job", path = "../../pkg/job/util" } util-mm = { package = "rivet-util-mm", path = "../../pkg/mm/util" } util-nsfw = { package = "rivet-util-nsfw", path = "../../pkg/nsfw/util" } @@ -64,7 +63,7 @@ cloud-namespace-token-development-create = { path = "../../pkg/cloud/ops/namespa cloud-namespace-token-public-create = { path = "../../pkg/cloud/ops/namespace-token-public-create" } cloud-version-get = { path = "../../pkg/cloud/ops/version-get" } cloud-version-publish = { path = "../../pkg/cloud/ops/version-publish" } -cluster-datacenter-list = { path = "../../pkg/cluster/ops/datacenter-list" } +cluster = { path = "../../pkg/cluster" } custom-user-avatar-list-for-game = { path = "../../pkg/custom-user-avatar/ops/list-for-game" } custom-user-avatar-upload-complete = { path = "../../pkg/custom-user-avatar/ops/upload-complete" } game-banner-upload-complete = { path = "../../pkg/game/ops/banner-upload-complete" } diff --git a/svc/api/cloud/src/route/tiers.rs b/svc/api/cloud/src/route/tiers.rs index d299dd542..29a41acc0 100644 --- a/svc/api/cloud/src/route/tiers.rs +++ b/svc/api/cloud/src/route/tiers.rs @@ -10,14 +10,17 @@ pub async fn list_tiers( ctx: Ctx, _watch_index: WatchIndexQuery, ) -> GlobalResult { - let datacenters_res = op!([ctx] cluster_datacenter_list { - cluster_ids: vec![util_cluster::default_cluster_id().into()], - }) - .await?; + let datacenters_res = ctx.op(cluster::ops::datacenter::list::Input { + cluster_ids: vec![cluster::util::default_cluster_id()], + }).await?; let cluster = unwrap!(datacenters_res.clusters.first()); let res = op!([ctx] tier_list { - region_ids: cluster.datacenter_ids.clone(), + region_ids: cluster.datacenter_ids + .iter() + .cloned() + .map(Into::into) + .collect::>(), }) .await?; diff --git a/svc/api/provision/Cargo.toml b/svc/api/provision/Cargo.toml index c5a862320..56e4bdb2b 100644 --- a/svc/api/provision/Cargo.toml +++ b/svc/api/provision/Cargo.toml @@ -28,10 +28,6 @@ tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } url = "2.2.2" uuid = { version = "1", features = ["v4"] } -util-cluster = { package = "rivet-util-cluster", path = "../../pkg/cluster/util" } -cluster-datacenter-get = { path = "../../pkg/cluster/ops/datacenter-get" } -cluster-datacenter-tls-get = { path = "../../pkg/cluster/ops/datacenter-tls-get" } -cluster-server-get = { path = "../../pkg/cluster/ops/server-get" } -cluster-server-resolve-for-ip = { path = "../../pkg/cluster/ops/server-resolve-for-ip" } +cluster = { path = "../../pkg/cluster" } diff --git a/svc/api/provision/src/route/datacenters.rs b/svc/api/provision/src/route/datacenters.rs index 6e5985138..51e914163 100644 --- a/svc/api/provision/src/route/datacenters.rs +++ b/svc/api/provision/src/route/datacenters.rs @@ -12,10 +12,11 @@ pub async fn tls( ) -> GlobalResult { ctx.auth().server()?; - let datacenter_res = op!([ctx] cluster_datacenter_tls_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; + let datacenter_res = ctx + .op(cluster::ops::datacenter::tls_get::Input { + datacenter_ids: vec![datacenter_id], + }) + .await?; let datacenter = unwrap!(datacenter_res.datacenters.first()); let (Some(job_cert_pem), Some(job_private_key_pem)) = diff --git a/svc/api/provision/src/route/servers.rs b/svc/api/provision/src/route/servers.rs index 4f8daa991..e4766b68d 100644 --- a/svc/api/provision/src/route/servers.rs +++ b/svc/api/provision/src/route/servers.rs @@ -1,7 +1,6 @@ use std::net::Ipv4Addr; use api_helper::{anchor::WatchIndexQuery, ctx::Ctx}; -use proto::backend; use rivet_api::models; use rivet_operation::prelude::*; @@ -16,41 +15,43 @@ pub async fn info( ctx.auth().server()?; // Find server based on public ip - let servers_res = op!([ctx] cluster_server_resolve_for_ip { - ips: vec![public_ip.to_string()], - }) - .await?; - - let server = unwrap!(servers_res.servers.first(), "server not found"); - let server_id = unwrap!(server.server_id); + let servers_res = ctx + .op(cluster::ops::server::resolve_for_ip::Input { + ips: vec![public_ip], + // We include destroyed because this request can only ever come from a running server. This means + // the server was marked as destroyed, but is still provisioning. + include_destroyed: true, + }) + .await?; + let server = unwrap!(servers_res.servers.first(), "server with ip not found"); // Get server info - let server_res = op!([ctx] cluster_server_get { - server_ids: vec![server_id], - }) - .await?; + let server_res = ctx + .op(cluster::ops::server::get::Input { + server_ids: vec![server.server_id], + }) + .await?; let server = unwrap!(server_res.servers.first(), "server not found"); // Get datacenter info - let datacenter_id = unwrap!(server.datacenter_id); - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id], - }) - .await?; + let datacenter_res = ctx + .op(cluster::ops::datacenter::get::Input { + datacenter_ids: vec![server.datacenter_id], + }) + .await?; let datacenter = unwrap!(datacenter_res.datacenters.first()); - let pool_type = unwrap!(backend::cluster::PoolType::from_i32(server.pool_type)); - let name = util_cluster::server_name( + let name = cluster::util::server_name( &datacenter.provider_datacenter_id, - pool_type, - server_id.as_uuid(), + server.pool_type.clone(), + server.server_id, ); Ok(models::ProvisionServersGetInfoResponse { name, - server_id: server_id.as_uuid(), - datacenter_id: datacenter_id.as_uuid(), - cluster_id: unwrap_ref!(datacenter.cluster_id).as_uuid(), - vlan_ip: unwrap_ref!(server.vlan_ip, "server should have vlan ip by now").clone(), + server_id: server.server_id, + datacenter_id: server.datacenter_id, + cluster_id: datacenter.cluster_id, + vlan_ip: unwrap_ref!(server.vlan_ip, "server should have vlan ip by now").to_string(), }) } diff --git a/svc/api/traefik-provider/Cargo.toml b/svc/api/traefik-provider/Cargo.toml index 69708ac00..391695a0c 100644 --- a/svc/api/traefik-provider/Cargo.toml +++ b/svc/api/traefik-provider/Cargo.toml @@ -37,7 +37,7 @@ util-cdn = { package = "rivet-util-cdn", path = "../../pkg/cdn/util" } util-job = { package = "rivet-util-job", path = "../../pkg/job/util" } uuid = { version = "1", features = ["v4"] } -cluster-server-list = { path = "../../pkg/cluster/ops/server-list" } +cluster = { path = "../../pkg/cluster" } [dev-dependencies] rivet-connection = { path = "../../../lib/connection" } diff --git a/svc/api/traefik-provider/src/route/tunnel.rs b/svc/api/traefik-provider/src/route/tunnel.rs index b38be7d4b..3b06ccd77 100644 --- a/svc/api/traefik-provider/src/route/tunnel.rs +++ b/svc/api/traefik-provider/src/route/tunnel.rs @@ -1,5 +1,4 @@ use api_helper::{anchor::WatchIndexQuery, ctx::Ctx}; -use proto::backend; use rivet_operation::prelude::*; use serde::{Deserialize, Serialize}; @@ -43,19 +42,21 @@ pub async fn build_ip_allowlist( ctx: &Ctx, config: &mut types::TraefikConfigResponse, ) -> GlobalResult<()> { - let servers_res = op!([ctx] cluster_server_list { - filter: Some(backend::cluster::ServerFilter { - pool_types: vec![backend::cluster::PoolType::Gg as i32], - ..Default::default() - }), - include_destroyed: false, - }) - .await?; + let servers_res = ctx + .op(cluster::ops::server::list::Input { + filter: cluster::types::Filter { + pool_types: Some(vec![cluster::types::PoolType::Gg]), + ..Default::default() + }, + include_destroyed: false, + }) + .await?; let public_ips = servers_res .servers .iter() - .filter_map(|server| server.public_ip.clone()) + .filter_map(|server| server.public_ip) + .map(|ip| ip.to_string()) .collect::>(); config.tcp.middlewares.insert( diff --git a/svc/pkg/cluster/Cargo.toml b/svc/pkg/cluster/Cargo.toml new file mode 100644 index 000000000..2927d751e --- /dev/null +++ b/svc/pkg/cluster/Cargo.toml @@ -0,0 +1,45 @@ +[package] +name = "cluster" +version = "0.0.1" +edition = "2018" +authors = ["Rivet Gaming, LLC "] +license = "Apache-2.0" + +[dependencies] +acme-lib = "0.9" +anyhow = "1.0" +chirp-workflow = { path = "../../../lib/chirp-workflow/core" } +cloudflare = "0.10.1" +http = "0.2" +include_dir = "0.7.3" +indoc = "1.0" +lazy_static = "1.4" +nomad-util = { path = "../../../lib/nomad-util" } +rand = "0.8" +rivet-metrics = { path = "../../../lib/metrics" } +rivet-operation = { path = "../../../lib/operation/core" } +rivet-runtime = { path = "../../../lib/runtime" } +s3-util = { path = "../../../lib/s3-util" } +serde = { version = "1.0.198", features = ["derive"] } +ssh2 = "0.9.4" +thiserror = "1.0" +trust-dns-resolver = { version = "0.23.2", features = ["dns-over-native-tls"] } + +ip-info = { path = "../ip/ops/info" } +linode = { path = "../linode" } +token-create = { path = "../token/ops/create" } + +[dependencies.nomad_client] +git = "https://github.com/rivet-gg/nomad-client" +rev = "abb66bf0c30c7ff5b0c695dae952481c33e538b5" # pragma: allowlist secret + +[dependencies.sqlx] +git = "https://github.com/rivet-gg/sqlx" +rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" +default-features = false +features = [ "json", "ipnetwork" ] + +[build-dependencies] +merkle_hash = "3.6" +hex = "0.4" +tokio = { version = "1.29", features = ["full"] } diff --git a/svc/pkg/cluster/worker/Service.toml b/svc/pkg/cluster/Service.toml similarity index 78% rename from svc/pkg/cluster/worker/Service.toml rename to svc/pkg/cluster/Service.toml index d4fb93058..c4e73d04c 100644 --- a/svc/pkg/cluster/worker/Service.toml +++ b/svc/pkg/cluster/Service.toml @@ -1,10 +1,10 @@ [service] -name = "cluster-worker" +name = "cluster" [runtime] kind = "rust" -[consumer] +[package] [secrets] "rivet/api_traefik_provider/token" = {} @@ -12,4 +12,4 @@ kind = "rust" "ssh/server/private_key_openssh" = {} [databases] -bucket-build = {} +db-cluster = {} diff --git a/svc/pkg/cluster/util/build.rs b/svc/pkg/cluster/build.rs similarity index 81% rename from svc/pkg/cluster/util/build.rs rename to svc/pkg/cluster/build.rs index eed68bebb..d454c7c22 100644 --- a/svc/pkg/cluster/util/build.rs +++ b/svc/pkg/cluster/build.rs @@ -1,7 +1,7 @@ use std::path::PathBuf; -use tokio::fs; use merkle_hash::MerkleTree; +use tokio::fs; // NOTE: This only gets the hash of the folder. Any template variables changed in the install scripts // will not update the hash. @@ -10,15 +10,11 @@ use merkle_hash::MerkleTree; async fn main() { let out_dir = PathBuf::from(std::env::var("OUT_DIR").unwrap()); let current_dir = std::env::current_dir().unwrap(); - let server_install_path = { - let mut dir = current_dir.clone(); - dir.pop(); - - dir.join("worker") - .join("src") - .join("workers") - .join("server_install") - }; + let server_install_path = current_dir + .join("src") + .join("workflows") + .join("server") + .join("install"); // Add rereun statement println!("cargo:rerun-if-changed={}", server_install_path.display()); diff --git a/svc/pkg/cluster/db/cluster/migrations/20240701225245_add_json.down.sql b/svc/pkg/cluster/db/cluster/migrations/20240701225245_add_json.down.sql new file mode 100644 index 000000000..e69de29bb diff --git a/svc/pkg/cluster/db/cluster/migrations/20240701225245_add_json.up.sql b/svc/pkg/cluster/db/cluster/migrations/20240701225245_add_json.up.sql new file mode 100644 index 000000000..8995966b8 --- /dev/null +++ b/svc/pkg/cluster/db/cluster/migrations/20240701225245_add_json.up.sql @@ -0,0 +1,27 @@ +ALTER TABLE datacenters + ADD COLUMN pools2 JSONB, -- Vec + ADD COLUMN provider2 JSONB, -- cluster::types::Provider + ADD COLUMN build_delivery_method2 JSONB; -- cluster::types::BuildDeliveryMethod + +ALTER TABLE servers + ADD COLUMN pool_type2 JSONB; -- cluster::types::PoolType + +-- No longer needed +DROP TABLE server_images_linode; + +ALTER TABLE datacenter_tls + ADD COLUMN state2 JSONB; -- cluster::types::TlsState + +CREATE TABLE server_images2 ( + provider JSONB, + install_hash TEXT, + datacenter_id UUID, + pool_type JSONB, + + create_ts INT NOT NULL, + -- After the image expires and is destroyed + destroy_ts INT, + provider_image_id TEXT, + + PRIMARY KEY (provider, install_hash, datacenter_id, pool_type) +); diff --git a/svc/pkg/cluster/ops/datacenter-get/Cargo.toml b/svc/pkg/cluster/ops/datacenter-get/Cargo.toml deleted file mode 100644 index 7e1073ba2..000000000 --- a/svc/pkg/cluster/ops/datacenter-get/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-datacenter-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/datacenter-get/Service.toml b/svc/pkg/cluster/ops/datacenter-get/Service.toml deleted file mode 100644 index a0f9d3cb5..000000000 --- a/svc/pkg/cluster/ops/datacenter-get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-datacenter-get" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/datacenter-get/src/lib.rs b/svc/pkg/cluster/ops/datacenter-get/src/lib.rs deleted file mode 100644 index d0cd01826..000000000 --- a/svc/pkg/cluster/ops/datacenter-get/src/lib.rs +++ /dev/null @@ -1,107 +0,0 @@ -use std::convert::{TryFrom, TryInto}; - -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Datacenter { - datacenter_id: Uuid, - cluster_id: Uuid, - name_id: String, - display_name: String, - provider: i64, - provider_datacenter_id: String, - provider_api_token: Option, - pools: Vec, - build_delivery_method: i64, - prebakes_enabled: bool, - create_ts: i64, -} - -impl TryFrom for backend::cluster::Datacenter { - type Error = GlobalError; - - fn try_from(value: Datacenter) -> GlobalResult { - let pools = cluster::msg::datacenter_create::Pools::decode(value.pools.as_slice())?.pools; - - Ok(backend::cluster::Datacenter { - datacenter_id: Some(value.datacenter_id.into()), - cluster_id: Some(value.cluster_id.into()), - name_id: value.name_id, - display_name: value.display_name, - create_ts: value.create_ts, - provider: value.provider as i32, - provider_datacenter_id: value.provider_datacenter_id, - provider_api_token: value.provider_api_token, - pools, - build_delivery_method: value.build_delivery_method as i32, - prebakes_enabled: value.prebakes_enabled, - }) - } -} - -#[operation(name = "cluster-datacenter-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let datacenter_ids = ctx - .datacenter_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let datacenters = ctx - .cache() - .fetch_all_proto("cluster.datacenters", datacenter_ids, { - let ctx = ctx.base(); - move |mut cache, datacenter_ids| { - let ctx = ctx.clone(); - async move { - let dcs = get_dcs(ctx, datacenter_ids).await?; - for dc in dcs { - let dc_id = unwrap!(dc.datacenter_id).as_uuid(); - cache.resolve(&dc_id, dc); - } - - Ok(cache) - } - } - }) - .await?; - - Ok(cluster::datacenter_get::Response { datacenters }) -} - -async fn get_dcs( - ctx: OperationContext<()>, - datacenter_ids: Vec, -) -> GlobalResult> { - let configs = sql_fetch_all!( - [ctx, Datacenter] - " - SELECT - datacenter_id, - cluster_id, - name_id, - display_name, - provider, - provider_datacenter_id, - provider_api_token, - pools, - build_delivery_method, - prebakes_enabled, - create_ts - FROM db_cluster.datacenters - WHERE datacenter_id = ANY($1) - ", - datacenter_ids, - ) - .await?; - - let datacenters = configs - .into_iter() - .map(TryInto::try_into) - .collect::>>()?; - - Ok(datacenters) -} diff --git a/svc/pkg/cluster/ops/datacenter-list/Cargo.toml b/svc/pkg/cluster/ops/datacenter-list/Cargo.toml deleted file mode 100644 index 9d8912e10..000000000 --- a/svc/pkg/cluster/ops/datacenter-list/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-datacenter-list" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/datacenter-list/Service.toml b/svc/pkg/cluster/ops/datacenter-list/Service.toml deleted file mode 100644 index ebad6361d..000000000 --- a/svc/pkg/cluster/ops/datacenter-list/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-datacenter-list" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/datacenter-list/src/lib.rs b/svc/pkg/cluster/ops/datacenter-list/src/lib.rs deleted file mode 100644 index 674e76562..000000000 --- a/svc/pkg/cluster/ops/datacenter-list/src/lib.rs +++ /dev/null @@ -1,62 +0,0 @@ -use std::collections::HashMap; - -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Datacenter { - cluster_id: Uuid, - datacenter_id: Uuid, -} - -#[operation(name = "cluster-datacenter-list")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let cluster_ids = ctx - .cluster_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let datacenters = sql_fetch_all!( - [ctx, Datacenter] - " - SELECT - cluster_id, - datacenter_id - FROM db_cluster.datacenters - WHERE cluster_id = ANY($1) - ", - &cluster_ids - ) - .await?; - - // Fill in empty clusters - let mut dcs_by_cluster_id = cluster_ids - .iter() - .map(|cluster_id| (*cluster_id, Vec::new())) - .collect::>>(); - - for dc in datacenters { - dcs_by_cluster_id - .entry(dc.cluster_id) - .or_default() - .push(dc.datacenter_id); - } - - Ok(cluster::datacenter_list::Response { - clusters: dcs_by_cluster_id - .into_iter() - .map( - |(cluster_id, datacenter_ids)| cluster::datacenter_list::response::Cluster { - cluster_id: Some(cluster_id.into()), - datacenter_ids: datacenter_ids - .into_iter() - .map(Into::into) - .collect::>(), - }, - ) - .collect::>(), - }) -} diff --git a/svc/pkg/cluster/ops/datacenter-location-get/Cargo.toml b/svc/pkg/cluster/ops/datacenter-location-get/Cargo.toml deleted file mode 100644 index b8121e63d..000000000 --- a/svc/pkg/cluster/ops/datacenter-location-get/Cargo.toml +++ /dev/null @@ -1,20 +0,0 @@ -[package] -name = "cluster-datacenter-location-get" -version = "0.0.1" -edition = "2021" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } - -ip-info = { path = "../../../ip/ops/info" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/datacenter-location-get/Service.toml b/svc/pkg/cluster/ops/datacenter-location-get/Service.toml deleted file mode 100644 index f6c3656b9..000000000 --- a/svc/pkg/cluster/ops/datacenter-location-get/Service.toml +++ /dev/null @@ -1,7 +0,0 @@ -[service] -name = "cluster-datacenter-location-get" - -[runtime] -kind = "rust" - -[operation] diff --git a/svc/pkg/cluster/ops/datacenter-location-get/src/lib.rs b/svc/pkg/cluster/ops/datacenter-location-get/src/lib.rs deleted file mode 100644 index 86ebbca11..000000000 --- a/svc/pkg/cluster/ops/datacenter-location-get/src/lib.rs +++ /dev/null @@ -1,100 +0,0 @@ -use std::net::IpAddr; - -use futures_util::{StreamExt, TryStreamExt}; -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[operation(name = "cluster-datacenter-location-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let datacenter_ids = ctx - .datacenter_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let datacenters = ctx - .cache() - .fetch_all_proto("cluster.datacenters.location", datacenter_ids, { - let ctx = ctx.base(); - move |mut cache, datacenter_ids| { - let ctx = ctx.clone(); - async move { - let dcs = query_dcs(ctx, datacenter_ids).await?; - for dc in dcs { - let dc_id = unwrap!(dc.datacenter_id).as_uuid(); - cache.resolve(&dc_id, dc); - } - - Ok(cache) - } - } - }) - .await?; - - Ok(cluster::datacenter_location_get::Response { datacenters }) -} - -async fn query_dcs( - ctx: OperationContext<()>, - datacenter_ids: Vec, -) -> GlobalResult> { - // NOTE: if there is no active GG node in a datacenter, we cannot retrieve its location - // Fetch the gg node public ip for each datacenter (there may be more than one, hence `DISTINCT`) - let server_rows = sql_fetch_all!( - [ctx, (Uuid, Option,)] - " - SELECT DISTINCT - datacenter_id, public_ip - FROM db_cluster.servers - WHERE - datacenter_id = ANY($1) AND - pool_type = $2 AND - cloud_destroy_ts IS NULL - -- For consistency - ORDER BY public_ip DESC - ", - &datacenter_ids, - backend::cluster::PoolType::Gg as i64, - ) - .await?; - - let coords_res = futures_util::stream::iter(server_rows) - .map(|(datacenter_id, public_ip)| { - let ctx = ctx.base(); - - async move { - if let Some(public_ip) = public_ip { - // Fetch IP info of GG node (this is cached inside `ip_info`) - let ip_info_res = op!([ctx] ip_info { - ip: public_ip.to_string(), - provider: ip::info::Provider::IpInfoIo as i32, - }) - .await?; - GlobalResult::Ok(( - datacenter_id, - ip_info_res - .ip_info - .as_ref() - .and_then(|info| info.coords.clone()), - )) - } else { - GlobalResult::Ok((datacenter_id, None)) - } - } - }) - .buffer_unordered(8) - .try_collect::>() - .await?; - - Ok(coords_res - .into_iter() - .map( - |(datacenter_id, coords)| cluster::datacenter_location_get::response::Datacenter { - datacenter_id: Some(datacenter_id.into()), - coords, - }, - ) - .collect::>()) -} diff --git a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Service.toml b/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Service.toml deleted file mode 100644 index aa845fc9a..000000000 --- a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-datacenter-resolve-for-name-id" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/src/lib.rs b/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/src/lib.rs deleted file mode 100644 index ff2e9e3ec..000000000 --- a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/src/lib.rs +++ /dev/null @@ -1,41 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Datacenter { - datacenter_id: Uuid, - name_id: String, -} - -#[operation(name = "cluster-datacenter-resolve-for-name-id")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let cluster_id = unwrap_ref!(ctx.cluster_id).as_uuid(); - - let datacenters = sql_fetch_all!( - [ctx, Datacenter] - " - SELECT - datacenter_id, - name_id - FROM db_cluster.datacenters - WHERE - cluster_id = $1 AND - name_id = ANY($2) - ", - &cluster_id, - &ctx.name_ids, - ) - .await? - .into_iter() - .map( - |dc| cluster::datacenter_resolve_for_name_id::response::Datacenter { - datacenter_id: Some(dc.datacenter_id.into()), - name_id: dc.name_id, - }, - ) - .collect::>(); - - Ok(cluster::datacenter_resolve_for_name_id::Response { datacenters }) -} diff --git a/svc/pkg/cluster/ops/datacenter-tls-get/Cargo.toml b/svc/pkg/cluster/ops/datacenter-tls-get/Cargo.toml deleted file mode 100644 index 9b3cd70b5..000000000 --- a/svc/pkg/cluster/ops/datacenter-tls-get/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-datacenter-tls-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/datacenter-tls-get/Service.toml b/svc/pkg/cluster/ops/datacenter-tls-get/Service.toml deleted file mode 100644 index a09426cfc..000000000 --- a/svc/pkg/cluster/ops/datacenter-tls-get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-datacenter-tls-get" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/datacenter-tls-get/src/lib.rs b/svc/pkg/cluster/ops/datacenter-tls-get/src/lib.rs deleted file mode 100644 index 3aaa8d074..000000000 --- a/svc/pkg/cluster/ops/datacenter-tls-get/src/lib.rs +++ /dev/null @@ -1,60 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct DatacenterTls { - datacenter_id: Uuid, - gg_cert_pem: Option, - gg_private_key_pem: Option, - job_cert_pem: Option, - job_private_key_pem: Option, - state: i64, - expire_ts: i64, -} - -impl From for cluster::datacenter_tls_get::response::Datacenter { - fn from(value: DatacenterTls) -> Self { - cluster::datacenter_tls_get::response::Datacenter { - datacenter_id: Some(value.datacenter_id.into()), - gg_cert_pem: value.gg_cert_pem, - gg_private_key_pem: value.gg_private_key_pem, - job_cert_pem: value.job_cert_pem, - job_private_key_pem: value.job_private_key_pem, - state: value.state as i32, - expire_ts: value.expire_ts, - } - } -} - -#[operation(name = "cluster-datacenter-tls-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let datacenter_ids = ctx - .datacenter_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let rows = sql_fetch_all!( - [ctx, DatacenterTls] - " - SELECT - datacenter_id, - gg_cert_pem, - gg_private_key_pem, - job_cert_pem, - job_private_key_pem, - state, - expire_ts - FROM db_cluster.datacenter_tls - WHERE datacenter_id = ANY($1) - ", - datacenter_ids, - ) - .await?; - - Ok(cluster::datacenter_tls_get::Response { - datacenters: rows.into_iter().map(Into::into).collect::>(), - }) -} diff --git a/svc/pkg/cluster/ops/datacenter-topology-get/Cargo.toml b/svc/pkg/cluster/ops/datacenter-topology-get/Cargo.toml deleted file mode 100644 index 51d851aa8..000000000 --- a/svc/pkg/cluster/ops/datacenter-topology-get/Cargo.toml +++ /dev/null @@ -1,25 +0,0 @@ -[package] -name = "cluster-datacenter-topology-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -lazy_static = "1.4" -nomad-util = { path = "../../../../../lib/nomad-util" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.nomad_client] -git = "https://github.com/rivet-gg/nomad-client" -rev = "abb66bf0c30c7ff5b0c695dae952481c33e538b5" # pragma: allowlist secret - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/datacenter-topology-get/README.md b/svc/pkg/cluster/ops/datacenter-topology-get/README.md deleted file mode 100644 index b24df2068..000000000 --- a/svc/pkg/cluster/ops/datacenter-topology-get/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# datacenter-topology-get - -Fetch the nomad topology for all job servers in a datacenter diff --git a/svc/pkg/cluster/ops/datacenter-topology-get/Service.toml b/svc/pkg/cluster/ops/datacenter-topology-get/Service.toml deleted file mode 100644 index 3c31348cf..000000000 --- a/svc/pkg/cluster/ops/datacenter-topology-get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-datacenter-topology-get" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/get-for-game/Cargo.toml b/svc/pkg/cluster/ops/get-for-game/Cargo.toml deleted file mode 100644 index 5ac4fb817..000000000 --- a/svc/pkg/cluster/ops/get-for-game/Cargo.toml +++ /dev/null @@ -1,20 +0,0 @@ -[package] -name = "cluster-get-for-game" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/get-for-game/Service.toml b/svc/pkg/cluster/ops/get-for-game/Service.toml deleted file mode 100644 index c6b8f6f34..000000000 --- a/svc/pkg/cluster/ops/get-for-game/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-get-for-game" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/get-for-game/src/lib.rs b/svc/pkg/cluster/ops/get-for-game/src/lib.rs deleted file mode 100644 index 053a09d0b..000000000 --- a/svc/pkg/cluster/ops/get-for-game/src/lib.rs +++ /dev/null @@ -1,42 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[operation(name = "cluster-get-for-game")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let game_ids = ctx - .game_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let rows = sql_fetch_optional!( - [ctx, (Uuid, Option)] - " - SELECT - g.game_id, gc.cluster_id - FROM unnest($1) AS g(game_id) - LEFT JOIN db_cluster.games AS gc - ON g.game_id = gc.game_id - ", - game_ids, - ) - .await?; - - Ok(cluster::get_for_game::Response { - games: rows - .into_iter() - .map( - |(game_id, cluster_id)| cluster::get_for_game::response::Game { - game_id: Some(game_id.into()), - cluster_id: Some( - cluster_id - .unwrap_or_else(util_cluster::default_cluster_id) - .into(), - ), - }, - ) - .collect::>(), - }) -} diff --git a/svc/pkg/cluster/ops/get-for-game/tests/integration.rs b/svc/pkg/cluster/ops/get-for-game/tests/integration.rs deleted file mode 100644 index f840758a6..000000000 --- a/svc/pkg/cluster/ops/get-for-game/tests/integration.rs +++ /dev/null @@ -1,24 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker_test] -async fn empty(ctx: TestCtx) { - let game_id = Uuid::new_v4(); - let cluster_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::game_link(game_id, cluster_id) -> cluster::msg::game_link_complete { - game_id: Some(game_id.into()), - cluster_id: Some(cluster_id.into()), - }) - .await - .unwrap(); - - let games_res = op!([ctx] cluster_get_for_game { - game_ids: vec![game_id.into()], - }) - .await - .unwrap(); - let game = games_res.games.first().unwrap(); - - assert_eq!(cluster_id, game.cluster_id.unwrap().as_uuid()); -} diff --git a/svc/pkg/cluster/ops/get/Cargo.toml b/svc/pkg/cluster/ops/get/Cargo.toml deleted file mode 100644 index 7f62318ae..000000000 --- a/svc/pkg/cluster/ops/get/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/get/Service.toml b/svc/pkg/cluster/ops/get/Service.toml deleted file mode 100644 index 06f53f69b..000000000 --- a/svc/pkg/cluster/ops/get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-get" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/get/src/lib.rs b/svc/pkg/cluster/ops/get/src/lib.rs deleted file mode 100644 index e4892bc35..000000000 --- a/svc/pkg/cluster/ops/get/src/lib.rs +++ /dev/null @@ -1,53 +0,0 @@ -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Cluster { - cluster_id: Uuid, - name_id: String, - owner_team_id: Option, - create_ts: i64, -} - -impl From for backend::cluster::Cluster { - fn from(value: Cluster) -> Self { - backend::cluster::Cluster { - cluster_id: Some(value.cluster_id.into()), - name_id: value.name_id, - owner_team_id: value.owner_team_id.map(Into::into), - create_ts: value.create_ts, - } - } -} - -#[operation(name = "cluster-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let crdb = ctx.crdb().await?; - let cluster_ids = ctx - .cluster_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let clusters = sql_fetch_all!( - [ctx, Cluster, &crdb] - " - SELECT - cluster_id, - name_id, - owner_team_id, - create_ts - FROM db_cluster.clusters - WHERE cluster_id = ANY($1) - ", - cluster_ids - ) - .await? - .into_iter() - .map(Into::into) - .collect::>(); - - Ok(cluster::get::Response { clusters }) -} diff --git a/svc/pkg/cluster/ops/get/tests/integration.rs b/svc/pkg/cluster/ops/get/tests/integration.rs deleted file mode 100644 index c273681e3..000000000 --- a/svc/pkg/cluster/ops/get/tests/integration.rs +++ /dev/null @@ -1,24 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker_test] -async fn empty(ctx: TestCtx) { - let cluster_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) - .await - .unwrap(); - - let res = op!([ctx] cluster_get { - cluster_ids: vec![cluster_id.into()], - }) - .await - .unwrap(); - let cluster = res.clusters.first().expect("cluster not found"); - - assert_eq!(cluster_id, cluster.cluster_id.unwrap().as_uuid()); -} diff --git a/svc/pkg/cluster/ops/list/Cargo.toml b/svc/pkg/cluster/ops/list/Cargo.toml deleted file mode 100644 index 99f578e4a..000000000 --- a/svc/pkg/cluster/ops/list/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-list" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/list/Service.toml b/svc/pkg/cluster/ops/list/Service.toml deleted file mode 100644 index a41334f3f..000000000 --- a/svc/pkg/cluster/ops/list/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-list" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/list/src/lib.rs b/svc/pkg/cluster/ops/list/src/lib.rs deleted file mode 100644 index 62cca800f..000000000 --- a/svc/pkg/cluster/ops/list/src/lib.rs +++ /dev/null @@ -1,46 +0,0 @@ -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Cluster { - cluster_id: Uuid, - name_id: String, - owner_team_id: Option, - create_ts: i64, -} - -impl From for backend::cluster::Cluster { - fn from(value: Cluster) -> Self { - backend::cluster::Cluster { - cluster_id: Some(value.cluster_id.into()), - name_id: value.name_id, - owner_team_id: value.owner_team_id.map(Into::into), - create_ts: value.create_ts, - } - } -} - -#[operation(name = "cluster-list")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let crdb = ctx.crdb().await?; - - let cluster_ids = sql_fetch_all!( - [ctx, Cluster, &crdb] - " - SELECT - cluster_id, - name_id, - owner_team_id, - create_ts - FROM db_cluster.clusters - ", - ) - .await? - .into_iter() - .map(|cluster| cluster.cluster_id.into()) - .collect::>(); - - Ok(cluster::list::Response { cluster_ids }) -} diff --git a/svc/pkg/cluster/ops/list/tests/integration.rs b/svc/pkg/cluster/ops/list/tests/integration.rs deleted file mode 100644 index 6090a9020..000000000 --- a/svc/pkg/cluster/ops/list/tests/integration.rs +++ /dev/null @@ -1,26 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker_test] -async fn list_single_cluster(ctx: TestCtx) { - let cluster_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) - .await - .unwrap(); - - let res = op!([ctx] cluster_list {}).await.unwrap(); - - // The cluster should be in the list of all clusters - let new_cluster_id = res - .cluster_ids - .into_iter() - .find(|id| id.as_uuid() == cluster_id) - .unwrap(); - - assert_eq!(cluster_id, new_cluster_id.as_uuid()); -} diff --git a/svc/pkg/cluster/ops/resolve-for-name-id/Cargo.toml b/svc/pkg/cluster/ops/resolve-for-name-id/Cargo.toml deleted file mode 100644 index 2fdac5937..000000000 --- a/svc/pkg/cluster/ops/resolve-for-name-id/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-resolve-for-name-id" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/resolve-for-name-id/Service.toml b/svc/pkg/cluster/ops/resolve-for-name-id/Service.toml deleted file mode 100644 index e944c42f5..000000000 --- a/svc/pkg/cluster/ops/resolve-for-name-id/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-resolve-for-name-id" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/resolve-for-name-id/src/lib.rs b/svc/pkg/cluster/ops/resolve-for-name-id/src/lib.rs deleted file mode 100644 index 8cafcf105..000000000 --- a/svc/pkg/cluster/ops/resolve-for-name-id/src/lib.rs +++ /dev/null @@ -1,35 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Cluster { - cluster_id: Uuid, - name_id: String, -} - -#[operation(name = "cluster-resolve-for-name-id")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let clusters = sql_fetch_all!( - [ctx, Cluster] - " - SELECT - cluster_id, - name_id - FROM db_cluster.clusters - WHERE - name_id = ANY($1) - ", - &ctx.name_ids, - ) - .await? - .into_iter() - .map(|dc| cluster::resolve_for_name_id::response::Cluster { - cluster_id: Some(dc.cluster_id.into()), - name_id: dc.name_id, - }) - .collect::>(); - - Ok(cluster::resolve_for_name_id::Response { clusters }) -} diff --git a/svc/pkg/cluster/ops/server-destroy-with-filter/Cargo.toml b/svc/pkg/cluster/ops/server-destroy-with-filter/Cargo.toml deleted file mode 100644 index d2e39e60b..000000000 --- a/svc/pkg/cluster/ops/server-destroy-with-filter/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-server-destroy-with-filter" -version = "0.0.1" -edition = "2021" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -cluster-server-list = { path = "../server-list" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/server-destroy-with-filter/Service.toml b/svc/pkg/cluster/ops/server-destroy-with-filter/Service.toml deleted file mode 100644 index 12aba4dd1..000000000 --- a/svc/pkg/cluster/ops/server-destroy-with-filter/Service.toml +++ /dev/null @@ -1,7 +0,0 @@ -[service] -name = "cluster-server-destroy-with-filter" - -[runtime] -kind = "rust" - -[operation] diff --git a/svc/pkg/cluster/ops/server-destroy-with-filter/src/lib.rs b/svc/pkg/cluster/ops/server-destroy-with-filter/src/lib.rs deleted file mode 100644 index 7e2a367ca..000000000 --- a/svc/pkg/cluster/ops/server-destroy-with-filter/src/lib.rs +++ /dev/null @@ -1,57 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; -use std::collections::HashSet; - -#[operation(name = "cluster-server-destroy-with-filter")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let servers_res = op!([ctx] cluster_server_list { - filter: ctx.filter.clone(), - }) - .await?; - - // Flag as destroyed - let server_ids = servers_res - .servers - .iter() - .filter_map(|x| x.server_id) - .map(|x| x.as_uuid()) - .collect::>(); - sql_execute!( - [ctx] - " - UPDATE db_cluster.servers - SET cloud_destroy_ts = $2 - WHERE server_id = ANY($1) - ", - &server_ids, - util::timestamp::now(), - ) - .await?; - - // Destroy server - for server_id in &server_ids { - msg!([ctx] cluster::msg::server_destroy(server_id) { - server_id: Some(server_id.clone().into()), - force: false, - }) - .await?; - } - - // Trigger scale event - let dc_ids = servers_res - .servers - .iter() - .filter_map(|x| x.datacenter_id) - .map(|x| x.as_uuid()) - .collect::>(); - for dc_id in dc_ids { - msg!([ctx] cluster::msg::datacenter_scale(dc_id) { - datacenter_id: Some(dc_id.into()), - }) - .await?; - } - - Ok(cluster::server_destroy_with_filter::Response {}) -} diff --git a/svc/pkg/cluster/ops/server-destroy-with-filter/tests/integration.rs b/svc/pkg/cluster/ops/server-destroy-with-filter/tests/integration.rs deleted file mode 100644 index d7d641e21..000000000 --- a/svc/pkg/cluster/ops/server-destroy-with-filter/tests/integration.rs +++ /dev/null @@ -1,6 +0,0 @@ -use chirp_worker::prelude::*; - -#[worker_test] -async fn basic(ctx: TestCtx) { - // TODO: -} diff --git a/svc/pkg/cluster/ops/server-get/Cargo.toml b/svc/pkg/cluster/ops/server-get/Cargo.toml deleted file mode 100644 index 5861543d2..000000000 --- a/svc/pkg/cluster/ops/server-get/Cargo.toml +++ /dev/null @@ -1,20 +0,0 @@ -[package] -name = "cluster-server-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false -features = [ "ipnetwork" ] - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/server-get/Service.toml b/svc/pkg/cluster/ops/server-get/Service.toml deleted file mode 100644 index 496afacad..000000000 --- a/svc/pkg/cluster/ops/server-get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-server-get" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/server-get/src/lib.rs b/svc/pkg/cluster/ops/server-get/src/lib.rs deleted file mode 100644 index ff7297cfb..000000000 --- a/svc/pkg/cluster/ops/server-get/src/lib.rs +++ /dev/null @@ -1,71 +0,0 @@ -use std::{ - convert::{TryFrom, TryInto}, - net::IpAddr, -}; - -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Server { - server_id: Uuid, - cluster_id: Uuid, - datacenter_id: Uuid, - pool_type: i64, - vlan_ip: Option, - public_ip: Option, - cloud_destroy_ts: Option, -} - -impl TryFrom for backend::cluster::Server { - type Error = GlobalError; - - fn try_from(value: Server) -> GlobalResult { - Ok(backend::cluster::Server { - server_id: Some(value.server_id.into()), - cluster_id: Some(value.cluster_id.into()), - datacenter_id: Some(value.datacenter_id.into()), - pool_type: value.pool_type.try_into()?, - vlan_ip: value.vlan_ip.map(|ip| ip.to_string()), - public_ip: value.public_ip.map(|ip| ip.to_string()), - cloud_destroy_ts: value.cloud_destroy_ts, - }) - } -} - -#[operation(name = "cluster-server-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let server_ids = ctx - .server_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); - - let servers = sql_fetch_all!( - [ctx, Server] - " - SELECT - server_id, - d.cluster_id, - s.datacenter_id, - pool_type, - vlan_ip, - public_ip, - cloud_destroy_ts - FROM db_cluster.servers AS s - LEFT JOIN db_cluster.datacenters AS d ON s.datacenter_id = d.datacenter_id - WHERE server_id = ANY($1) - ", - server_ids - ) - .await?; - - Ok(cluster::server_get::Response { - servers: servers - .into_iter() - .map(TryInto::try_into) - .collect::>>()?, - }) -} diff --git a/svc/pkg/cluster/ops/server-get/tests/integration.rs b/svc/pkg/cluster/ops/server-get/tests/integration.rs deleted file mode 100644 index d2939f861..000000000 --- a/svc/pkg/cluster/ops/server-get/tests/integration.rs +++ /dev/null @@ -1,6 +0,0 @@ -use chirp_worker::prelude::*; - -#[worker_test] -async fn empty(_ctx: TestCtx) { - // TODO: -} diff --git a/svc/pkg/cluster/ops/server-list/Cargo.toml b/svc/pkg/cluster/ops/server-list/Cargo.toml deleted file mode 100644 index e79c7aabf..000000000 --- a/svc/pkg/cluster/ops/server-list/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-server-list" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/server-list/Service.toml b/svc/pkg/cluster/ops/server-list/Service.toml deleted file mode 100644 index f0def326f..000000000 --- a/svc/pkg/cluster/ops/server-list/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-server-list" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/server-list/src/lib.rs b/svc/pkg/cluster/ops/server-list/src/lib.rs deleted file mode 100644 index cc9b50d91..000000000 --- a/svc/pkg/cluster/ops/server-list/src/lib.rs +++ /dev/null @@ -1,123 +0,0 @@ -use std::{ - convert::{TryFrom, TryInto}, - net::IpAddr, -}; - -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Server { - server_id: Uuid, - cluster_id: Uuid, - datacenter_id: Uuid, - pool_type: i64, - vlan_ip: Option, - public_ip: Option, - cloud_destroy_ts: Option, -} - -impl TryFrom for backend::cluster::Server { - type Error = GlobalError; - - fn try_from(value: Server) -> GlobalResult { - Ok(backend::cluster::Server { - server_id: Some(value.server_id.into()), - cluster_id: Some(value.cluster_id.into()), - datacenter_id: Some(value.datacenter_id.into()), - pool_type: value.pool_type.try_into()?, - vlan_ip: value.vlan_ip.map(|ip| ip.to_string()), - public_ip: value.public_ip.map(|ip| ip.to_string()), - cloud_destroy_ts: value.cloud_destroy_ts, - }) - } -} - -#[operation(name = "cluster-server-list")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let filter = unwrap_ref!(ctx.filter); - - let server_ids = if filter.filter_server_ids { - Some( - filter - .server_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let cluster_ids = if filter.filter_cluster_ids { - Some( - filter - .cluster_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let datacenter_ids = if filter.filter_datacenter_ids { - Some( - filter - .datacenter_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let pool_types = if filter.filter_pool_types { - Some(&filter.pool_types) - } else { - None - }; - let public_ips = if filter.filter_public_ips { - Some(&filter.public_ips) - } else { - None - }; - - let servers = sql_fetch_all!( - [ctx, Server] - " - SELECT - s.server_id, - d.cluster_id, - s.datacenter_id, - s.pool_type, - s.vlan_ip, - s.public_ip, - s.cloud_destroy_ts - FROM db_cluster.servers AS s - JOIN db_cluster.datacenters AS d - ON s.datacenter_id = d.datacenter_id - WHERE - ($1 OR s.cloud_destroy_ts IS NULL) - AND ($2 IS NULL OR s.server_id = ANY($2)) - AND ($3 IS NULL OR d.cluster_id = ANY($3)) - AND ($4 IS NULL OR s.datacenter_id = ANY($4)) - AND ($5 IS NULL OR s.pool_type = ANY($5)) - AND ($6 IS NULL OR s.public_ip = ANY($6::inet[])) - ", - ctx.include_destroyed, - &server_ids, - &cluster_ids, - &datacenter_ids, - &pool_types, - &public_ips, - ) - .await?; - - Ok(cluster::server_list::Response { - servers: servers - .into_iter() - .map(TryInto::try_into) - .collect::>>()?, - }) -} diff --git a/svc/pkg/cluster/ops/server-list/tests/integration.rs b/svc/pkg/cluster/ops/server-list/tests/integration.rs deleted file mode 100644 index d2939f861..000000000 --- a/svc/pkg/cluster/ops/server-list/tests/integration.rs +++ /dev/null @@ -1,6 +0,0 @@ -use chirp_worker::prelude::*; - -#[worker_test] -async fn empty(_ctx: TestCtx) { - // TODO: -} diff --git a/svc/pkg/cluster/ops/server-resolve-for-ip/Cargo.toml b/svc/pkg/cluster/ops/server-resolve-for-ip/Cargo.toml deleted file mode 100644 index 43e8d63f7..000000000 --- a/svc/pkg/cluster/ops/server-resolve-for-ip/Cargo.toml +++ /dev/null @@ -1,19 +0,0 @@ -[package] -name = "cluster-server-resolve-for-ip" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/ops/server-resolve-for-ip/Service.toml b/svc/pkg/cluster/ops/server-resolve-for-ip/Service.toml deleted file mode 100644 index 0ad9fa42d..000000000 --- a/svc/pkg/cluster/ops/server-resolve-for-ip/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "cluster-server-resolve-for-ip" - -[runtime] -kind = "rust" - -[operation] - -[databases] -db-cluster = {} diff --git a/svc/pkg/cluster/ops/server-resolve-for-ip/src/lib.rs b/svc/pkg/cluster/ops/server-resolve-for-ip/src/lib.rs deleted file mode 100644 index d28efd5a2..000000000 --- a/svc/pkg/cluster/ops/server-resolve-for-ip/src/lib.rs +++ /dev/null @@ -1,42 +0,0 @@ -use std::net::IpAddr; - -use proto::backend::pkg::*; -use rivet_operation::prelude::*; - -#[derive(sqlx::FromRow)] -struct Server { - server_id: Uuid, - public_ip: IpAddr, -} - -impl From for cluster::server_resolve_for_ip::response::Server { - fn from(value: Server) -> Self { - cluster::server_resolve_for_ip::response::Server { - server_id: Some(value.server_id.into()), - public_ip: value.public_ip.to_string(), - } - } -} - -#[operation(name = "cluster-server-resolve-for-ip")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let servers = sql_fetch_all!( - [ctx, Server] - " - SELECT - server_id, public_ip - FROM db_cluster.servers - WHERE - public_ip = ANY($1) AND - cloud_destroy_ts IS NULL - ", - &ctx.ips - ) - .await?; - - Ok(cluster::server_resolve_for_ip::Response { - servers: servers.into_iter().map(Into::into).collect::>(), - }) -} diff --git a/svc/pkg/cluster/ops/server-resolve-for-ip/tests/integration.rs b/svc/pkg/cluster/ops/server-resolve-for-ip/tests/integration.rs deleted file mode 100644 index d2939f861..000000000 --- a/svc/pkg/cluster/ops/server-resolve-for-ip/tests/integration.rs +++ /dev/null @@ -1,6 +0,0 @@ -use chirp_worker::prelude::*; - -#[worker_test] -async fn empty(_ctx: TestCtx) { - // TODO: -} diff --git a/svc/pkg/cluster/proto/datacenter-get.proto b/svc/pkg/cluster/proto/datacenter-get.proto deleted file mode 100644 index eb992858a..000000000 --- a/svc/pkg/cluster/proto/datacenter-get.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_get; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated rivet.common.Uuid datacenter_ids = 1; -} - -message Response { - repeated rivet.backend.cluster.Datacenter datacenters = 1; -} diff --git a/svc/pkg/cluster/proto/datacenter-list.proto b/svc/pkg/cluster/proto/datacenter-list.proto deleted file mode 100644 index c774f704f..000000000 --- a/svc/pkg/cluster/proto/datacenter-list.proto +++ /dev/null @@ -1,19 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_list; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated rivet.common.Uuid cluster_ids = 1; -} - -message Response { - message Cluster { - rivet.common.Uuid cluster_id = 1; - repeated rivet.common.Uuid datacenter_ids = 2; - } - - repeated Cluster clusters = 1; -} diff --git a/svc/pkg/cluster/proto/datacenter-location-get.proto b/svc/pkg/cluster/proto/datacenter-location-get.proto deleted file mode 100644 index 6b2395398..000000000 --- a/svc/pkg/cluster/proto/datacenter-location-get.proto +++ /dev/null @@ -1,19 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_location_get; - -import "proto/common.proto"; -import "proto/backend/net.proto"; - -message Request { - repeated rivet.common.Uuid datacenter_ids = 1; -} - -message Response { - message Datacenter { - rivet.common.Uuid datacenter_id = 1; - rivet.backend.net.Coordinates coords = 2; - } - - repeated Datacenter datacenters = 1; -} diff --git a/svc/pkg/cluster/proto/datacenter-resolve-for-name-id.proto b/svc/pkg/cluster/proto/datacenter-resolve-for-name-id.proto deleted file mode 100644 index 6390d1cbe..000000000 --- a/svc/pkg/cluster/proto/datacenter-resolve-for-name-id.proto +++ /dev/null @@ -1,20 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_resolve_for_name_id; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - rivet.common.Uuid cluster_id = 1; - repeated string name_ids = 2; -} - -message Response { - message Datacenter { - rivet.common.Uuid datacenter_id = 1; - string name_id = 2; - } - - repeated Datacenter datacenters = 1; -} diff --git a/svc/pkg/cluster/proto/datacenter-tls-get.proto b/svc/pkg/cluster/proto/datacenter-tls-get.proto deleted file mode 100644 index c43abcd89..000000000 --- a/svc/pkg/cluster/proto/datacenter-tls-get.proto +++ /dev/null @@ -1,24 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_tls_get; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated rivet.common.Uuid datacenter_ids = 1; -} - -message Response { - message Datacenter { - rivet.common.Uuid datacenter_id = 1; - optional string gg_cert_pem = 2; - optional string gg_private_key_pem = 3; - optional string job_cert_pem = 4; - optional string job_private_key_pem = 5; - rivet.backend.cluster.TlsState state = 6; - int64 expire_ts = 7; - } - - repeated Datacenter datacenters = 1; -} diff --git a/svc/pkg/cluster/proto/datacenter-topology-get.proto b/svc/pkg/cluster/proto/datacenter-topology-get.proto deleted file mode 100644 index be9a47862..000000000 --- a/svc/pkg/cluster/proto/datacenter-topology-get.proto +++ /dev/null @@ -1,32 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.datacenter_topology_get; - -import "proto/common.proto"; - -message Request { - repeated rivet.common.Uuid datacenter_ids = 1; -} - -message Response { - message Server { - rivet.common.Uuid server_id = 1; - string node_id = 2; - - Stats usage = 3; - Stats limits = 4; - } - - message Datacenter { - rivet.common.Uuid datacenter_id = 1; - repeated Server servers = 2; - } - - message Stats { - uint64 cpu = 1; // mhz - uint64 memory = 2; // mb - uint64 disk = 3; // mb - } - - repeated Datacenter datacenters = 1; -} diff --git a/svc/pkg/cluster/proto/get-for-game.proto b/svc/pkg/cluster/proto/get-for-game.proto deleted file mode 100644 index 211b4ab20..000000000 --- a/svc/pkg/cluster/proto/get-for-game.proto +++ /dev/null @@ -1,18 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.get_for_game; - -import "proto/common.proto"; - -message Request { - repeated rivet.common.Uuid game_ids = 1; -} - -message Response { - message Game { - rivet.common.Uuid game_id = 1; - rivet.common.Uuid cluster_id = 2; - } - - repeated Game games = 1; -} diff --git a/svc/pkg/cluster/proto/get.proto b/svc/pkg/cluster/proto/get.proto deleted file mode 100644 index 3d995108a..000000000 --- a/svc/pkg/cluster/proto/get.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.get; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated rivet.common.Uuid cluster_ids = 1; -} - -message Response { - repeated rivet.backend.cluster.Cluster clusters = 1; -} diff --git a/svc/pkg/cluster/proto/list.proto b/svc/pkg/cluster/proto/list.proto deleted file mode 100644 index f05ea35cc..000000000 --- a/svc/pkg/cluster/proto/list.proto +++ /dev/null @@ -1,12 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.list; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request {} - -message Response { - repeated rivet.common.Uuid cluster_ids = 1; -} diff --git a/svc/pkg/cluster/proto/msg/create-complete.proto b/svc/pkg/cluster/proto/msg/create-complete.proto deleted file mode 100644 index cd62379ca..000000000 --- a/svc/pkg/cluster/proto/msg/create-complete.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.create_complete; - -import "proto/common.proto"; - -/// name = "msg-cluster-create-complete" -/// parameters = [ -/// { name = "cluster_id" }, -/// ] -message Message { - rivet.common.Uuid cluster_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/create.proto b/svc/pkg/cluster/proto/msg/create.proto deleted file mode 100644 index 2c24142cc..000000000 --- a/svc/pkg/cluster/proto/msg/create.proto +++ /dev/null @@ -1,15 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.create; - -import "proto/common.proto"; - -/// name = "msg-cluster-create" -/// parameters = [ -/// { name = "cluster_id" }, -/// ] -message Message { - rivet.common.Uuid cluster_id = 1; - string name_id = 2; - optional rivet.common.Uuid owner_team_id = 3; -} diff --git a/svc/pkg/cluster/proto/msg/datacenter-create.proto b/svc/pkg/cluster/proto/msg/datacenter-create.proto deleted file mode 100644 index dbac3a646..000000000 --- a/svc/pkg/cluster/proto/msg/datacenter-create.proto +++ /dev/null @@ -1,30 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.datacenter_create; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-datacenter-create" -/// parameters = [ -/// { name = "datacenter_id" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; - rivet.common.Uuid cluster_id = 2; - string name_id = 3; - string display_name = 4; - - rivet.backend.cluster.Provider provider = 5; - string provider_datacenter_id = 6; - optional string provider_api_token = 7; - - repeated rivet.backend.cluster.Pool pools = 8; - rivet.backend.cluster.BuildDeliveryMethod build_delivery_method = 9; - bool prebakes_enabled = 10; -} - -// Helper proto for writing to sql -message Pools { - repeated rivet.backend.cluster.Pool pools = 1; -} diff --git a/svc/pkg/cluster/proto/msg/datacenter-scale.proto b/svc/pkg/cluster/proto/msg/datacenter-scale.proto deleted file mode 100644 index cbe7a468a..000000000 --- a/svc/pkg/cluster/proto/msg/datacenter-scale.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.datacenter_scale; - -import "proto/common.proto"; - -/// name = "msg-cluster-datacenter-scale" -/// parameters = [ -/// { name = "datacenter_id" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/datacenter-tls-issue.proto b/svc/pkg/cluster/proto/msg/datacenter-tls-issue.proto deleted file mode 100644 index 3272f7cf1..000000000 --- a/svc/pkg/cluster/proto/msg/datacenter-tls-issue.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.datacenter_tls_issue; - -import "proto/common.proto"; - -/// name = "msg-cluster-datacenter-tls-issue" -/// parameters = [ -/// { name = "datacenter_id" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; - bool renew = 2; -} diff --git a/svc/pkg/cluster/proto/msg/datacenter-update.proto b/svc/pkg/cluster/proto/msg/datacenter-update.proto deleted file mode 100644 index 1fa69eb15..000000000 --- a/svc/pkg/cluster/proto/msg/datacenter-update.proto +++ /dev/null @@ -1,27 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.datacenter_update; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-datacenter-update" -/// parameters = [ -/// { name = "datacenter_id" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; - repeated PoolUpdate pools = 2; - optional bool prebakes_enabled = 3; -} - -message PoolUpdate { - rivet.backend.cluster.PoolType pool_type = 1; - - // Each can be optionally updated - repeated rivet.backend.cluster.Hardware hardware = 2; - optional uint32 desired_count = 3; - optional uint32 min_count = 6; - optional uint32 max_count = 4; - optional uint64 drain_timeout = 5; -} diff --git a/svc/pkg/cluster/proto/msg/game-link-complete.proto b/svc/pkg/cluster/proto/msg/game-link-complete.proto deleted file mode 100644 index 2c55e17b0..000000000 --- a/svc/pkg/cluster/proto/msg/game-link-complete.proto +++ /dev/null @@ -1,15 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.game_link_complete; - -import "proto/common.proto"; - -/// name = "msg-cluster-game-link-complete" -/// parameters = [ -/// { name = "game_id" }, -/// { name = "cluster_id" }, -/// ] -message Message { - rivet.common.Uuid game_id = 1; - rivet.common.Uuid cluster_id = 2; -} diff --git a/svc/pkg/cluster/proto/msg/game-link.proto b/svc/pkg/cluster/proto/msg/game-link.proto deleted file mode 100644 index b59b6decb..000000000 --- a/svc/pkg/cluster/proto/msg/game-link.proto +++ /dev/null @@ -1,15 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.game_link; - -import "proto/common.proto"; - -/// name = "msg-cluster-game-link" -/// parameters = [ -/// { name = "game_id" }, -/// { name = "cluster_id" }, -/// ] -message Message { - rivet.common.Uuid game_id = 1; - rivet.common.Uuid cluster_id = 2; -} diff --git a/svc/pkg/cluster/proto/msg/server-destroy-complete.proto b/svc/pkg/cluster/proto/msg/server-destroy-complete.proto deleted file mode 100644 index a580ab1dc..000000000 --- a/svc/pkg/cluster/proto/msg/server-destroy-complete.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_destroy_complete; - -import "proto/common.proto"; - -/// name = "msg-cluster-server-destroy-complete" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/server-destroy.proto b/svc/pkg/cluster/proto/msg/server-destroy.proto deleted file mode 100644 index 7e65d0447..000000000 --- a/svc/pkg/cluster/proto/msg/server-destroy.proto +++ /dev/null @@ -1,15 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_destroy; - -import "proto/common.proto"; - -/// name = "msg-cluster-server-destroy" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; - // Destroys the server even if it isn't fully provisioned yet - bool force = 2; -} diff --git a/svc/pkg/cluster/proto/msg/server-dns-create.proto b/svc/pkg/cluster/proto/msg/server-dns-create.proto deleted file mode 100644 index 2a19067c2..000000000 --- a/svc/pkg/cluster/proto/msg/server-dns-create.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_dns_create; - -import "proto/common.proto"; - -/// name = "msg-cluster-server-dns-create" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/server-dns-delete.proto b/svc/pkg/cluster/proto/msg/server-dns-delete.proto deleted file mode 100644 index 63786e8d5..000000000 --- a/svc/pkg/cluster/proto/msg/server-dns-delete.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_dns_delete; - -import "proto/common.proto"; - -/// name = "msg-cluster-server-dns-delete" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/server-drain.proto b/svc/pkg/cluster/proto/msg/server-drain.proto deleted file mode 100644 index b4770d0d8..000000000 --- a/svc/pkg/cluster/proto/msg/server-drain.proto +++ /dev/null @@ -1,13 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_drain; - -import "proto/common.proto"; - -/// name = "msg-cluster-server-drain" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; -} diff --git a/svc/pkg/cluster/proto/msg/server-install-complete.proto b/svc/pkg/cluster/proto/msg/server-install-complete.proto deleted file mode 100644 index b15d698bc..000000000 --- a/svc/pkg/cluster/proto/msg/server-install-complete.proto +++ /dev/null @@ -1,20 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_install_complete; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-server-install-complete" -/// parameters = [ -/// { name = "request_id" }, -/// ] -message Message { - rivet.common.Uuid request_id = 1; - - string public_ip = 2; - rivet.common.Uuid datacenter_id = 3; - // If set in server install message - optional rivet.common.Uuid server_id = 4; - rivet.backend.cluster.Provider provider = 5; -} diff --git a/svc/pkg/cluster/proto/msg/server-install.proto b/svc/pkg/cluster/proto/msg/server-install.proto deleted file mode 100644 index a061c2665..000000000 --- a/svc/pkg/cluster/proto/msg/server-install.proto +++ /dev/null @@ -1,26 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_install; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-server-install" -/// parameters = [ -/// { name = "request_id" }, -/// ] -message Message { - rivet.common.Uuid request_id = 1; - - string public_ip = 2; - rivet.backend.cluster.PoolType pool_type = 3; - - rivet.common.Uuid datacenter_id = 4; - // Unset when installing prebake servers since they don't have an id. Used to check if - // the server is currently being deleted to prevent installation - optional rivet.common.Uuid server_id = 5; - - // Simply passed to the install complete message - rivet.backend.cluster.Provider provider = 6; - bool initialize_immediately = 7; -} diff --git a/svc/pkg/cluster/proto/msg/server-provision.proto b/svc/pkg/cluster/proto/msg/server-provision.proto deleted file mode 100644 index faa3106b2..000000000 --- a/svc/pkg/cluster/proto/msg/server-provision.proto +++ /dev/null @@ -1,18 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_provision; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-server-provision" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; - rivet.common.Uuid server_id = 2; - rivet.backend.cluster.PoolType pool_type = 3; - rivet.backend.cluster.Provider provider = 4; - repeated string tags = 5; -} diff --git a/svc/pkg/cluster/proto/msg/server-taint.proto b/svc/pkg/cluster/proto/msg/server-taint.proto deleted file mode 100644 index 34513f39f..000000000 --- a/svc/pkg/cluster/proto/msg/server-taint.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_taint; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-server-taint" -/// parameters = [ -/// { name = "request_id" }, -/// ] -message Message { - rivet.backend.cluster.ServerFilter filter = 2; -} diff --git a/svc/pkg/cluster/proto/msg/server-undrain.proto b/svc/pkg/cluster/proto/msg/server-undrain.proto deleted file mode 100644 index cc8feb152..000000000 --- a/svc/pkg/cluster/proto/msg/server-undrain.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.msg.server_undrain; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-cluster-server-undrain" -/// parameters = [ -/// { name = "server_id" }, -/// ] -message Message { - rivet.common.Uuid server_id = 1; -} diff --git a/svc/pkg/cluster/proto/resolve-for-name-id.proto b/svc/pkg/cluster/proto/resolve-for-name-id.proto deleted file mode 100644 index 41bc28bef..000000000 --- a/svc/pkg/cluster/proto/resolve-for-name-id.proto +++ /dev/null @@ -1,19 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.resolve_for_name_id; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated string name_ids = 1; -} - -message Response { - message Cluster { - rivet.common.Uuid cluster_id = 1; - string name_id = 2; - } - - repeated Cluster clusters = 1; -} diff --git a/svc/pkg/cluster/proto/server-destroy-with-filter.proto b/svc/pkg/cluster/proto/server-destroy-with-filter.proto deleted file mode 100644 index 7dec0b41c..000000000 --- a/svc/pkg/cluster/proto/server-destroy-with-filter.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.server_destroy_with_filter; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - rivet.backend.cluster.ServerFilter filter = 1; -} - -message Response { - -} diff --git a/svc/pkg/cluster/proto/server-get.proto b/svc/pkg/cluster/proto/server-get.proto deleted file mode 100644 index a994fd3dd..000000000 --- a/svc/pkg/cluster/proto/server-get.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.server_get; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated rivet.common.Uuid server_ids = 1; -} - -message Response { - repeated rivet.backend.cluster.Server servers = 1; -} diff --git a/svc/pkg/cluster/proto/server-list.proto b/svc/pkg/cluster/proto/server-list.proto deleted file mode 100644 index 297a7f5cf..000000000 --- a/svc/pkg/cluster/proto/server-list.proto +++ /dev/null @@ -1,15 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.server_list; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - rivet.backend.cluster.ServerFilter filter = 1; - bool include_destroyed = 2; -} - -message Response { - repeated rivet.backend.cluster.Server servers = 1; -} diff --git a/svc/pkg/cluster/proto/server-resolve-for-ip.proto b/svc/pkg/cluster/proto/server-resolve-for-ip.proto deleted file mode 100644 index 19bd25ee5..000000000 --- a/svc/pkg/cluster/proto/server-resolve-for-ip.proto +++ /dev/null @@ -1,19 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.cluster.server_resolve_for_ip; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - repeated string ips = 1; -} - -message Response { - message Server { - string public_ip = 1; - rivet.common.Uuid server_id = 2; - } - - repeated Server servers = 1; -} diff --git a/svc/pkg/cluster/src/lib.rs b/svc/pkg/cluster/src/lib.rs new file mode 100644 index 000000000..b2ed2a6e7 --- /dev/null +++ b/svc/pkg/cluster/src/lib.rs @@ -0,0 +1,24 @@ +use chirp_workflow::prelude::*; + +pub mod ops; +pub mod types; +pub mod util; +pub mod workflows; + +pub fn registry() -> Registry { + use workflows::*; + + let mut registry = Registry::new(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + registry.register_workflow::(); + + registry +} diff --git a/svc/pkg/cluster/src/ops/datacenter/get.rs b/svc/pkg/cluster/src/ops/datacenter/get.rs new file mode 100644 index 000000000..21eec564f --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/get.rs @@ -0,0 +1,131 @@ +use std::convert::{TryFrom, TryInto}; + +use chirp_workflow::prelude::*; +use rivet_operation::prelude::{proto::backend, Message}; + +use crate::types::{BuildDeliveryMethod, Datacenter, Pool, Provider}; + +#[derive(Debug)] +pub struct Input { + pub datacenter_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub datacenters: Vec, +} + +#[derive(sqlx::FromRow)] +struct DatacenterRow { + datacenter_id: Uuid, + cluster_id: Uuid, + name_id: String, + display_name: String, + provider2: Option>, + provider: i64, + provider_datacenter_id: String, + provider_api_token: Option, + pools2: Option>>, + pools: Vec, + build_delivery_method2: Option>, + build_delivery_method: i64, + prebakes_enabled: bool, + create_ts: i64, +} + +impl TryFrom for Datacenter { + type Error = GlobalError; + + fn try_from(value: DatacenterRow) -> GlobalResult { + Ok(Datacenter { + datacenter_id: value.datacenter_id, + cluster_id: value.cluster_id, + name_id: value.name_id, + display_name: value.display_name, + create_ts: value.create_ts, + // Handle backwards compatibility + provider: if let Some(provider) = value.provider2 { + provider.0 + } else { + value.provider.try_into()? + }, + provider_datacenter_id: value.provider_datacenter_id, + provider_api_token: value.provider_api_token, + // Handle backwards compatibility + pools: if let Some(pools) = value.pools2 { + pools.0 + } else { + let proto = backend::cluster::Pools::decode(value.pools.as_slice())?.pools; + + proto + .into_iter() + .map(TryInto::try_into) + .collect::>>()? + }, + // Handle backwards compatibility + build_delivery_method: if let Some(build_delivery_method) = value.build_delivery_method2 + { + build_delivery_method.0 + } else { + value.build_delivery_method.try_into()? + }, + prebakes_enabled: value.prebakes_enabled, + }) + } +} + +#[operation] +pub async fn cluster_datacenter_get(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let datacenters = ctx + .cache() + .fetch_all_json("cluster.datacenters", input.datacenter_ids.clone(), { + let ctx = ctx.clone(); + move |mut cache, datacenter_ids| { + let ctx = ctx.clone(); + async move { + let dcs = get_dcs(ctx, datacenter_ids).await?; + for dc in dcs { + let dc_id = dc.datacenter_id; + cache.resolve(&dc_id, dc); + } + + Ok(cache) + } + } + }) + .await?; + + Ok(Output { datacenters }) +} + +async fn get_dcs(ctx: OperationCtx, datacenter_ids: Vec) -> GlobalResult> { + let dc_rows = sql_fetch_all!( + [ctx, DatacenterRow] + " + SELECT + datacenter_id, + cluster_id, + name_id, + display_name, + provider, + provider2, + provider_datacenter_id, + provider_api_token, + pools, + pools2, + build_delivery_method, + build_delivery_method2, + prebakes_enabled, + create_ts + FROM db_cluster.datacenters + WHERE datacenter_id = ANY($1) + ", + datacenter_ids, + ) + .await?; + + dc_rows + .into_iter() + .map(TryInto::try_into) + .collect::>>() +} diff --git a/svc/pkg/cluster/src/ops/datacenter/list.rs b/svc/pkg/cluster/src/ops/datacenter/list.rs new file mode 100644 index 000000000..405a98c84 --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/list.rs @@ -0,0 +1,59 @@ +use std::collections::HashMap; + +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input { + pub cluster_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub clusters: Vec, +} + +#[derive(Debug)] +pub struct Cluster { + pub cluster_id: Uuid, + pub datacenter_ids: Vec, +} + +#[operation] +pub async fn cluster_datacenter_list(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let rows = sql_fetch_all!( + [ctx, (Uuid, Uuid)] + " + SELECT + cluster_id, + datacenter_id + FROM db_cluster.datacenters + WHERE cluster_id = ANY($1) + ", + &input.cluster_ids, + ) + .await?; + + // Fill in empty clusters + let mut dcs_by_cluster_id = input + .cluster_ids + .iter() + .map(|cluster_id| (*cluster_id, Vec::new())) + .collect::>>(); + + for (cluster_id, datacenter_id) in rows { + dcs_by_cluster_id + .entry(cluster_id) + .or_default() + .push(datacenter_id); + } + + Ok(Output { + clusters: dcs_by_cluster_id + .into_iter() + .map(|(cluster_id, datacenter_ids)| Cluster { + cluster_id, + datacenter_ids, + }) + .collect::>(), + }) +} diff --git a/svc/pkg/cluster/src/ops/datacenter/location_get.rs b/svc/pkg/cluster/src/ops/datacenter/location_get.rs new file mode 100644 index 000000000..627683970 --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/location_get.rs @@ -0,0 +1,123 @@ +use std::net::IpAddr; + +use chirp_workflow::prelude::*; +use futures_util::{StreamExt, TryStreamExt}; +use rivet_operation::prelude::proto::backend::pkg::*; + +use crate::types::PoolType; + +#[derive(Debug)] +pub struct Input { + pub datacenter_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub datacenters: Vec, +} + +#[derive(Debug, Serialize, Deserialize)] +pub struct Datacenter { + pub datacenter_id: Uuid, + pub coords: Coordinates, +} + +// TODO: Move to a common types lib +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct Coordinates { + pub longitude: f64, + pub latitude: f64, +} + +#[operation] +pub async fn cluster_datacenter_location_get( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let datacenters = ctx + .cache() + .fetch_all_json( + "cluster.datacenters.location", + input.datacenter_ids.clone(), + { + let ctx = ctx.clone(); + move |mut cache, datacenter_ids| { + let ctx = ctx.clone(); + async move { + let dcs = query_dcs(ctx, datacenter_ids).await?; + for dc in dcs { + let dc_id = dc.datacenter_id; + cache.resolve(&dc_id, dc); + } + + Ok(cache) + } + } + }, + ) + .await?; + + Ok(Output { datacenters }) +} + +async fn query_dcs(ctx: OperationCtx, datacenter_ids: Vec) -> GlobalResult> { + // NOTE: if there is no active GG node in a datacenter, we cannot retrieve its location + // Fetch the gg node public ip for each datacenter (there may be more than one, hence `DISTINCT`) + let server_rows = sql_fetch_all!( + [ctx, (Uuid, IpAddr)] + " + SELECT DISTINCT + datacenter_id, public_ip + FROM db_cluster.servers + WHERE + datacenter_id = ANY($1) AND + pool_type2 = $2 AND + public_ip IS NOT NULL AND + cloud_destroy_ts IS NULL + -- For consistency + ORDER BY public_ip DESC + ", + &datacenter_ids, + serde_json::to_string(&PoolType::Gg)?, + ) + .await?; + + let coords_res = futures_util::stream::iter(server_rows) + .map(|(datacenter_id, public_ip)| { + let ctx = ctx.clone(); + + async move { + // Fetch IP info of GG node (this is cached inside `ip_info`) + let ip_info_res = op!([ctx] ip_info { + ip: public_ip.to_string(), + provider: ip::info::Provider::IpInfoIo as i32, + }) + .await?; + + GlobalResult::Ok(( + datacenter_id, + ip_info_res + .ip_info + .as_ref() + .and_then(|info| info.coords.as_ref()) + .map(|coords| Coordinates { + longitude: coords.longitude, + latitude: coords.latitude, + }), + )) + } + }) + .buffer_unordered(8) + .try_collect::>() + .await?; + + Ok(coords_res + .into_iter() + .filter_map(|(datacenter_id, coords)| { + coords.map(|coords| Datacenter { + datacenter_id, + coords, + }) + }) + .collect::>()) +} diff --git a/svc/pkg/cluster/src/ops/datacenter/mod.rs b/svc/pkg/cluster/src/ops/datacenter/mod.rs new file mode 100644 index 000000000..dce3767e2 --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/mod.rs @@ -0,0 +1,6 @@ +pub mod get; +pub mod list; +pub mod location_get; +pub mod resolve_for_name_id; +pub mod tls_get; +pub mod topology_get; diff --git a/svc/pkg/cluster/src/ops/datacenter/resolve_for_name_id.rs b/svc/pkg/cluster/src/ops/datacenter/resolve_for_name_id.rs new file mode 100644 index 000000000..29905c0e4 --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/resolve_for_name_id.rs @@ -0,0 +1,42 @@ +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input { + pub cluster_id: Uuid, + pub name_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub datacenters: Vec, +} + +#[derive(Debug, sqlx::FromRow)] +pub struct Datacenter { + pub datacenter_id: Uuid, + pub name_id: String, +} + +#[operation] +pub async fn cluster_datacenter_resolve_for_name_id( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let datacenters = sql_fetch_all!( + [ctx, Datacenter] + " + SELECT + datacenter_id, + name_id + FROM db_cluster.datacenters + WHERE + cluster_id = $1 AND + name_id = ANY($2) + ", + &input.cluster_id, + &input.name_ids, + ) + .await?; + + Ok(Output { datacenters }) +} diff --git a/svc/pkg/cluster/src/ops/datacenter/tls_get.rs b/svc/pkg/cluster/src/ops/datacenter/tls_get.rs new file mode 100644 index 000000000..6315190f4 --- /dev/null +++ b/svc/pkg/cluster/src/ops/datacenter/tls_get.rs @@ -0,0 +1,88 @@ +use std::convert::{TryFrom, TryInto}; + +use chirp_workflow::prelude::*; + +use crate::types::TlsState; + +#[derive(sqlx::FromRow)] +struct DatacenterTlsRow { + datacenter_id: Uuid, + gg_cert_pem: Option, + gg_private_key_pem: Option, + job_cert_pem: Option, + job_private_key_pem: Option, + state: i64, + state2: Option>, + expire_ts: i64, +} + +impl TryFrom for DatacenterTls { + type Error = GlobalError; + + fn try_from(value: DatacenterTlsRow) -> GlobalResult { + Ok(DatacenterTls { + datacenter_id: value.datacenter_id, + gg_cert_pem: value.gg_cert_pem, + gg_private_key_pem: value.gg_private_key_pem, + job_cert_pem: value.job_cert_pem, + job_private_key_pem: value.job_private_key_pem, + // Handle backwards compatibility + state: if let Some(state) = value.state2 { + state.0 + } else { + value.state.try_into()? + }, + expire_ts: value.expire_ts, + }) + } +} + +#[derive(Debug)] +pub struct Input { + pub datacenter_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub datacenters: Vec, +} + +#[derive(Debug)] +pub struct DatacenterTls { + pub datacenter_id: Uuid, + pub gg_cert_pem: Option, + pub gg_private_key_pem: Option, + pub job_cert_pem: Option, + pub job_private_key_pem: Option, + pub state: TlsState, + pub expire_ts: i64, +} + +#[operation] +pub async fn cluster_datacenter_tls_get(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let rows = sql_fetch_all!( + [ctx, DatacenterTlsRow] + " + SELECT + datacenter_id, + gg_cert_pem, + gg_private_key_pem, + job_cert_pem, + job_private_key_pem, + state, + state2, + expire_ts + FROM db_cluster.datacenter_tls + WHERE datacenter_id = ANY($1) + ", + &input.datacenter_ids, + ) + .await?; + + Ok(Output { + datacenters: rows + .into_iter() + .map(TryInto::try_into) + .collect::>>()?, + }) +} diff --git a/svc/pkg/cluster/ops/datacenter-topology-get/src/lib.rs b/svc/pkg/cluster/src/ops/datacenter/topology_get.rs similarity index 75% rename from svc/pkg/cluster/ops/datacenter-topology-get/src/lib.rs rename to svc/pkg/cluster/src/ops/datacenter/topology_get.rs index 582a11ec1..5992d7987 100644 --- a/svc/pkg/cluster/ops/datacenter-topology-get/src/lib.rs +++ b/svc/pkg/cluster/src/ops/datacenter/topology_get.rs @@ -1,32 +1,57 @@ use std::collections::HashMap; +use chirp_workflow::prelude::*; use nomad_client::apis::{allocations_api, configuration::Configuration, nodes_api}; -use proto::backend::pkg::*; -use rivet_operation::prelude::*; lazy_static::lazy_static! { static ref NOMAD_CONFIG: Configuration = nomad_util::new_config_from_env().unwrap(); } #[derive(sqlx::FromRow)] -struct Server { +struct ServerRow { server_id: Uuid, datacenter_id: Uuid, nomad_node_id: String, } -#[operation(name = "cluster-datacenter-topology-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let datacenter_ids = ctx - .datacenter_ids - .iter() - .map(common::Uuid::as_uuid) - .collect::>(); +#[derive(Debug)] +pub struct Input { + pub datacenter_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub datacenters: Vec, +} + +#[derive(Debug)] +pub struct Datacenter { + pub datacenter_id: Uuid, + pub servers: Vec, +} + +#[derive(Debug)] +pub struct Server { + pub server_id: Uuid, + pub node_id: String, + pub usage: Stats, + pub limits: Stats, +} +#[derive(Debug)] +pub struct Stats { + pub cpu: u64, + pub memory: u64, + pub disk: u64, +} + +#[operation] +pub async fn cluster_datacenter_topology_get( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { let servers = sql_fetch_all!( - [ctx, Server] + [ctx, ServerRow] " SELECT server_id, datacenter_id, nomad_node_id @@ -37,7 +62,7 @@ pub async fn handle( cloud_destroy_ts IS NULL AND taint_ts IS NULL ", - &datacenter_ids, + &input.datacenter_ids, ) .await?; @@ -83,13 +108,14 @@ pub async fn handle( )?; // Preempt datacenters - let mut datacenters = datacenter_ids + let mut datacenters = input + .datacenter_ids .iter() .map(|datacenter_id| { ( *datacenter_id, - cluster::datacenter_topology_get::response::Datacenter { - datacenter_id: Some((*datacenter_id).into()), + Datacenter { + datacenter_id: *datacenter_id, servers: Vec::new(), }, ) @@ -97,7 +123,7 @@ pub async fn handle( .collect::>(); for server in servers { - let mut usage = cluster::datacenter_topology_get::response::Stats { + let mut usage = Stats { cpu: 0, memory: 0, disk: 0, @@ -146,7 +172,7 @@ pub async fn handle( format!("node not found {}", server.nomad_node_id) ); let resources = unwrap_ref!(node.node_resources); - let limits = cluster::datacenter_topology_get::response::Stats { + let limits = Stats { cpu: unwrap!(unwrap_ref!(resources.cpu).cpu_shares) as u64, memory: unwrap!(unwrap_ref!(resources.memory).memory_mb) as u64, disk: unwrap!(unwrap_ref!(resources.disk).disk_mb) as u64, @@ -154,17 +180,15 @@ pub async fn handle( let datacenter = unwrap!(datacenters.get_mut(&server.datacenter_id)); - datacenter - .servers - .push(cluster::datacenter_topology_get::response::Server { - server_id: Some(server.server_id.into()), - node_id: server.nomad_node_id, - usage: Some(usage), - limits: Some(limits), - }); + datacenter.servers.push(Server { + server_id: server.server_id, + node_id: server.nomad_node_id, + usage, + limits, + }); } - Ok(cluster::datacenter_topology_get::Response { - datacenters: datacenters.into_values().collect::>(), + Ok(Output { + datacenters: datacenters.into_values().collect(), }) } diff --git a/svc/pkg/cluster/src/ops/get.rs b/svc/pkg/cluster/src/ops/get.rs new file mode 100644 index 000000000..d4638e60c --- /dev/null +++ b/svc/pkg/cluster/src/ops/get.rs @@ -0,0 +1,33 @@ +use chirp_workflow::prelude::*; + +use crate::types::Cluster; + +#[derive(Debug)] +pub struct Input { + pub cluster_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub clusters: Vec, +} + +#[operation] +pub async fn cluster_get(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let clusters = sql_fetch_all!( + [ctx, Cluster] + " + SELECT + cluster_id, + name_id, + owner_team_id, + create_ts + FROM db_cluster.clusters + WHERE cluster_id = ANY($1) + ", + &input.cluster_ids, + ) + .await?; + + Ok(Output { clusters }) +} diff --git a/svc/pkg/cluster/src/ops/get_for_game.rs b/svc/pkg/cluster/src/ops/get_for_game.rs new file mode 100644 index 000000000..515de5bdc --- /dev/null +++ b/svc/pkg/cluster/src/ops/get_for_game.rs @@ -0,0 +1,43 @@ +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input { + pub game_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub games: Vec, +} + +#[derive(Debug)] +pub struct Game { + pub game_id: Uuid, + pub cluster_id: Uuid, +} + +#[operation] +pub async fn cluster_get_for_game(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let rows = sql_fetch_optional!( + [ctx, (Uuid, Option)] + " + SELECT + g.game_id, gc.cluster_id + FROM unnest($1) AS g(game_id) + LEFT JOIN db_cluster.games AS gc + ON g.game_id = gc.game_id + ", + &input.game_ids, + ) + .await?; + + Ok(Output { + games: rows + .into_iter() + .map(|(game_id, cluster_id)| Game { + game_id, + cluster_id: cluster_id.unwrap_or_else(crate::util::default_cluster_id), + }) + .collect::>(), + }) +} diff --git a/svc/pkg/cluster/src/ops/list.rs b/svc/pkg/cluster/src/ops/list.rs new file mode 100644 index 000000000..a2d853821 --- /dev/null +++ b/svc/pkg/cluster/src/ops/list.rs @@ -0,0 +1,26 @@ +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input {} + +#[derive(Debug)] +pub struct Output { + pub cluster_ids: Vec, +} + +#[operation] +pub async fn cluster_list(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let cluster_ids = sql_fetch_all!( + [ctx, (Uuid,)] + " + SELECT cluster_id + FROM db_cluster.clusters + ", + ) + .await? + .into_iter() + .map(|(cluster_id,)| cluster_id) + .collect::>(); + + Ok(Output { cluster_ids }) +} diff --git a/svc/pkg/cluster/src/ops/mod.rs b/svc/pkg/cluster/src/ops/mod.rs new file mode 100644 index 000000000..d69e65b3d --- /dev/null +++ b/svc/pkg/cluster/src/ops/mod.rs @@ -0,0 +1,6 @@ +pub mod datacenter; +pub mod get; +pub mod get_for_game; +pub mod list; +pub mod resolve_for_name_id; +pub mod server; diff --git a/svc/pkg/cluster/src/ops/resolve_for_name_id.rs b/svc/pkg/cluster/src/ops/resolve_for_name_id.rs new file mode 100644 index 000000000..bebb50684 --- /dev/null +++ b/svc/pkg/cluster/src/ops/resolve_for_name_id.rs @@ -0,0 +1,38 @@ +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input { + pub name_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub clusters: Vec, +} + +#[derive(Debug, sqlx::FromRow)] +pub struct Cluster { + pub cluster_id: Uuid, + pub name_id: String, +} + +#[operation] +pub async fn cluster_resolve_for_name_id( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let clusters = sql_fetch_all!( + [ctx, Cluster] + " + SELECT + cluster_id, + name_id + FROM db_cluster.clusters + WHERE name_id = ANY($1) + ", + &input.name_ids, + ) + .await?; + + Ok(Output { clusters }) +} diff --git a/svc/pkg/cluster/src/ops/server/destroy_with_filter.rs b/svc/pkg/cluster/src/ops/server/destroy_with_filter.rs new file mode 100644 index 000000000..e34f3119a --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/destroy_with_filter.rs @@ -0,0 +1,74 @@ +use std::collections::HashSet; + +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::types::Filter; + +#[derive(Debug)] +pub struct Input { + pub filter: Filter, +} + +#[derive(Debug)] +pub struct Output {} + +#[operation] +pub async fn cluster_server_destroy_with_filter( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let servers_res = ctx + .op(crate::ops::server::list::Input { + filter: input.filter.clone(), + include_destroyed: false, + }) + .await?; + + // Flag as destroyed + let server_ids = servers_res + .servers + .iter() + .map(|x| x.server_id) + .collect::>(); + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers + SET cloud_destroy_ts = $2 + WHERE server_id = ANY($1) + ", + &server_ids, + util::timestamp::now(), + ) + .await?; + + // Destroy servers + for server_id in server_ids { + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Destroy {}, + ) + .await?; + } + + // Trigger scale event + let dc_ids = servers_res + .servers + .iter() + .map(|x| x.datacenter_id) + .collect::>(); + for dc_id in dc_ids { + ctx.tagged_signal( + &json!({ + "datacenter_id": dc_id, + }), + crate::workflows::datacenter::Scale {}, + ) + .await?; + } + + Ok(Output {}) +} diff --git a/svc/pkg/cluster/src/ops/server/get.rs b/svc/pkg/cluster/src/ops/server/get.rs new file mode 100644 index 000000000..305cd9a00 --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/get.rs @@ -0,0 +1,75 @@ +use std::{ + convert::{TryFrom, TryInto}, + net::IpAddr, +}; + +use chirp_workflow::prelude::*; + +use crate::types::{PoolType, Server}; + +#[derive(Debug)] +pub struct Input { + pub server_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub servers: Vec, +} + +#[derive(sqlx::FromRow)] +pub(crate) struct ServerRow { + server_id: Uuid, + datacenter_id: Uuid, + pool_type2: Option>, + pool_type: i64, + vlan_ip: Option, + public_ip: Option, + cloud_destroy_ts: Option, +} + +impl TryFrom for Server { + type Error = GlobalError; + + fn try_from(value: ServerRow) -> GlobalResult { + Ok(Server { + server_id: value.server_id, + datacenter_id: value.datacenter_id, + // Handle backwards compatibility + pool_type: if let Some(pool_type) = value.pool_type2 { + pool_type.0 + } else { + value.pool_type.try_into()? + }, + vlan_ip: value.vlan_ip, + public_ip: value.public_ip, + cloud_destroy_ts: value.cloud_destroy_ts, + }) + } +} + +#[operation] +pub async fn cluster_server_get(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let servers = sql_fetch_all!( + [ctx, ServerRow] + " + SELECT + server_id, + datacenter_id, + pool_type, + pool_type2, + vlan_ip, + public_ip, + cloud_destroy_ts + FROM db_cluster.servers + WHERE server_id = ANY($1) + ", + &input.server_ids, + ) + .await? + .into_iter() + .map(TryInto::try_into) + .collect::>>()?; + + Ok(Output { servers }) +} diff --git a/svc/pkg/cluster/src/ops/server/list.rs b/svc/pkg/cluster/src/ops/server/list.rs new file mode 100644 index 000000000..2db0b3834 --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/list.rs @@ -0,0 +1,67 @@ +use std::{convert::TryInto, net::IpAddr}; + +use chirp_workflow::prelude::*; + +use super::get::ServerRow; +use crate::types::{Filter, Server}; + +#[derive(Debug)] +pub struct Input { + pub filter: Filter, + pub include_destroyed: bool, +} + +#[derive(Debug)] +pub struct Output { + pub servers: Vec, +} + +#[operation] +pub async fn cluster_server_list(ctx: &OperationCtx, input: &Input) -> GlobalResult { + let servers = sql_fetch_all!( + [ctx, ServerRow] + " + SELECT + s.server_id, + s.datacenter_id, + s.pool_type, + s.pool_type2, + s.vlan_ip, + s.public_ip, + s.cloud_destroy_ts + FROM db_cluster.servers AS s + JOIN db_cluster.datacenters AS d + ON s.datacenter_id = d.datacenter_id + WHERE + ($1 OR s.cloud_destroy_ts IS NULL) + AND ($2 IS NULL OR s.server_id = ANY($2)) + AND ($3 IS NULL OR s.datacenter_id = ANY($4)) + AND ($4 IS NULL OR d.cluster_id = ANY($3)) + AND ($5 IS NULL OR s.pool_type2 = ANY($5::JSONB[])) + AND ($6 IS NULL OR s.public_ip = ANY($6)) + ", + input.include_destroyed, + &input.filter.server_ids, + &input.filter.datacenter_ids, + &input.filter.cluster_ids, + input.filter.pool_types + .as_ref() + .map(|x| x.iter() + .map(serde_json::to_string) + .collect::, _>>() + ).transpose()?, + input.filter.public_ips + .as_ref() + .map(|x| x.iter() + .cloned() + .map(IpAddr::V4) + .collect::>() + ), + ) + .await? + .into_iter() + .map(TryInto::try_into) + .collect::>>()?; + + Ok(Output { servers }) +} diff --git a/svc/pkg/cluster/src/ops/server/mod.rs b/svc/pkg/cluster/src/ops/server/mod.rs new file mode 100644 index 000000000..78cccf9ac --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/mod.rs @@ -0,0 +1,5 @@ +pub mod destroy_with_filter; +pub mod get; +pub mod list; +pub mod resolve_for_ip; +pub mod taint_with_filter; diff --git a/svc/pkg/cluster/src/ops/server/resolve_for_ip.rs b/svc/pkg/cluster/src/ops/server/resolve_for_ip.rs new file mode 100644 index 000000000..4fd032b58 --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/resolve_for_ip.rs @@ -0,0 +1,46 @@ +use std::net::{IpAddr, Ipv4Addr}; + +use chirp_workflow::prelude::*; + +#[derive(Debug)] +pub struct Input { + pub ips: Vec, + pub include_destroyed: bool, +} + +#[derive(Debug)] +pub struct Output { + pub servers: Vec, +} + +#[derive(Debug, sqlx::FromRow)] +pub struct Server { + pub server_id: Uuid, + pub public_ip: IpAddr, +} + +#[operation] +pub async fn cluster_server_resolve_for_ip( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let servers = sql_fetch_all!( + [ctx, Server] + " + SELECT server_id, public_ip + FROM db_cluster.servers + WHERE + ($1 OR cloud_destroy_ts IS NULL) AND + public_ip = ANY($2) + ", + input.include_destroyed, + input.ips + .iter() + .cloned() + .map(IpAddr::V4) + .collect::>(), + ) + .await?; + + Ok(Output { servers }) +} diff --git a/svc/pkg/cluster/src/ops/server/taint_with_filter.rs b/svc/pkg/cluster/src/ops/server/taint_with_filter.rs new file mode 100644 index 000000000..802ac0114 --- /dev/null +++ b/svc/pkg/cluster/src/ops/server/taint_with_filter.rs @@ -0,0 +1,74 @@ +use std::collections::HashSet; + +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::types::Filter; + +#[derive(Debug)] +pub struct Input { + pub filter: Filter, +} + +#[derive(Debug)] +pub struct Output {} + +#[operation] +pub async fn cluster_server_taint_with_filter( + ctx: &OperationCtx, + input: &Input, +) -> GlobalResult { + let servers_res = ctx + .op(crate::ops::server::list::Input { + filter: input.filter.clone(), + include_destroyed: false, + }) + .await?; + + // Flag as tainted + let server_ids = servers_res + .servers + .iter() + .map(|x| x.server_id) + .collect::>(); + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers + SET taint_ts = $2 + WHERE server_id = ANY($1) + ", + &server_ids, + util::timestamp::now(), + ) + .await?; + + // Taint servers + for server_id in server_ids { + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Taint {}, + ) + .await?; + } + + // Trigger scale event + let dc_ids = servers_res + .servers + .iter() + .map(|x| x.datacenter_id) + .collect::>(); + for dc_id in dc_ids { + ctx.tagged_signal( + &json!({ + "datacenter_id": dc_id, + }), + crate::workflows::datacenter::Scale {}, + ) + .await?; + } + + Ok(Output {}) +} diff --git a/svc/pkg/cluster/src/types.rs b/svc/pkg/cluster/src/types.rs new file mode 100644 index 000000000..38c8e63e8 --- /dev/null +++ b/svc/pkg/cluster/src/types.rs @@ -0,0 +1,198 @@ +use std::{ + convert::{TryFrom, TryInto}, + net::{IpAddr, Ipv4Addr}, +}; + +use chirp_workflow::prelude::*; +use rivet_operation::prelude::proto::backend; +use serde::{Deserialize, Serialize}; + +#[derive(Debug, sqlx::FromRow)] +pub struct Cluster { + pub cluster_id: Uuid, + pub name_id: String, + /// Unset for the default cluster. + pub owner_team_id: Option, + pub create_ts: i64, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub struct Datacenter { + pub datacenter_id: Uuid, + pub cluster_id: Uuid, + pub name_id: String, + pub display_name: String, + pub provider: Provider, + pub provider_datacenter_id: String, + pub provider_api_token: Option, + pub pools: Vec, + pub build_delivery_method: BuildDeliveryMethod, + pub prebakes_enabled: bool, + pub create_ts: i64, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub enum Provider { + Linode, +} + +// Backwards compatibility +impl TryFrom for Provider { + type Error = GlobalError; + + fn try_from(value: i64) -> GlobalResult { + match value { + 0 => Ok(Provider::Linode), + _ => bail!("unexpected Provider variant"), + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub struct Pool { + pub pool_type: PoolType, + /// See docs on failover (/docs/packages/cluster/SERVER_PROVISIONING.md#creating-a-new-server) + pub hardware: Vec, + pub desired_count: u32, + pub min_count: u32, + pub max_count: u32, + pub drain_timeout: u64, +} + +// Backwards compatibility +impl TryFrom for Pool { + type Error = GlobalError; + + fn try_from(value: backend::cluster::Pool) -> GlobalResult { + Ok(Pool { + pool_type: (value.pool_type as i64).try_into()?, + hardware: value + .hardware + .iter() + .map(|h| Hardware { + provider_hardware: h.provider_hardware.clone(), + }) + .collect(), + desired_count: value.desired_count, + min_count: value.min_count, + max_count: value.max_count, + drain_timeout: value.drain_timeout, + }) + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash, PartialEq, Eq)] +pub enum PoolType { + Job, + Gg, + Ats, +} + +// Backwards compatibility +impl TryFrom for PoolType { + type Error = GlobalError; + + fn try_from(value: i64) -> GlobalResult { + match value { + 0 => Ok(PoolType::Job), + 1 => Ok(PoolType::Gg), + 2 => Ok(PoolType::Ats), + _ => bail!("unexpected PoolType variant"), + } + } +} +impl From for i64 { + fn from(value: PoolType) -> i64 { + match value { + PoolType::Job => 0, + PoolType::Gg => 1, + PoolType::Ats => 2, + } + } +} + +impl std::fmt::Display for PoolType { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + PoolType::Job => write!(f, "job"), + PoolType::Gg => write!(f, "gg"), + PoolType::Ats => write!(f, "ats"), + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub struct Hardware { + pub provider_hardware: String, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +pub struct PoolUpdate { + pub pool_type: PoolType, + + // Each can be optionally updated + pub hardware: Vec, + pub desired_count: Option, + pub min_count: Option, + pub max_count: Option, + pub drain_timeout: Option, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub enum BuildDeliveryMethod { + TrafficServer, + S3Direct, +} + +// Backwards compatibility +impl TryFrom for BuildDeliveryMethod { + type Error = GlobalError; + + fn try_from(value: i64) -> GlobalResult { + match value { + 0 => Ok(BuildDeliveryMethod::TrafficServer), + 1 => Ok(BuildDeliveryMethod::S3Direct), + _ => bail!("unexpected BuildDeliveryMethod variant"), + } + } +} + +#[derive(Debug)] +pub struct Server { + pub server_id: Uuid, + pub datacenter_id: Uuid, + pub pool_type: PoolType, + pub vlan_ip: Option, + pub public_ip: Option, + pub cloud_destroy_ts: Option, +} + +#[derive(Debug, Default, Clone)] +pub struct Filter { + pub server_ids: Option>, + pub datacenter_ids: Option>, + pub cluster_ids: Option>, + pub pool_types: Option>, + pub public_ips: Option>, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +pub enum TlsState { + Creating, + Active, + Renewing, +} + +// Backwards compatibility +impl TryFrom for TlsState { + type Error = GlobalError; + + fn try_from(value: i64) -> GlobalResult { + match value { + 0 => Ok(TlsState::Creating), + 1 => Ok(TlsState::Active), + 2 => Ok(TlsState::Renewing), + _ => bail!("unexpected TlsState variant"), + } + } +} diff --git a/svc/pkg/cluster/util/src/metrics.rs b/svc/pkg/cluster/src/util/metrics.rs similarity index 100% rename from svc/pkg/cluster/util/src/metrics.rs rename to svc/pkg/cluster/src/util/metrics.rs diff --git a/svc/pkg/cluster/src/util/mod.rs b/svc/pkg/cluster/src/util/mod.rs new file mode 100644 index 000000000..5256436cd --- /dev/null +++ b/svc/pkg/cluster/src/util/mod.rs @@ -0,0 +1,50 @@ +use chirp_workflow::prelude::*; +use cloudflare::framework as cf_framework; + +use crate::types::PoolType; + +pub mod metrics; +pub mod test; + +// Use the hash of the server install script in the image variant so that if the install scripts are updated +// we won't be using the old image anymore +pub const INSTALL_SCRIPT_HASH: &str = include_str!(concat!(env!("OUT_DIR"), "/hash.txt")); + +// TTL of the token written to prebake images. Prebake images are renewed before the token would expire +pub const SERVER_TOKEN_TTL: i64 = util::duration::days(30 * 6); + +#[derive(thiserror::Error, Debug)] +#[error("cloudflare: {source}")] +struct CloudflareError { + #[from] + source: anyhow::Error, +} + +// Cluster id for provisioning servers +pub fn default_cluster_id() -> Uuid { + Uuid::nil() +} + +pub fn server_name(provider_datacenter_id: &str, pool_type: PoolType, server_id: Uuid) -> String { + let ns = util::env::namespace(); + let pool_type_str = match pool_type { + PoolType::Job => "job", + PoolType::Gg => "gg", + PoolType::Ats => "ats", + }; + + format!("{ns}-{provider_datacenter_id}-{pool_type_str}-{server_id}",) +} + +pub(crate) async fn cf_client() -> GlobalResult { + // Create CF client + let cf_token = util::env::read_secret(&["cloudflare", "terraform", "auth_token"]).await?; + let client = cf_framework::async_api::Client::new( + cf_framework::auth::Credentials::UserAuthToken { token: cf_token }, + Default::default(), + cf_framework::Environment::Production, + ) + .map_err(CloudflareError::from)?; + + Ok(client) +} diff --git a/svc/pkg/cluster/util/src/test.rs b/svc/pkg/cluster/src/util/test.rs similarity index 100% rename from svc/pkg/cluster/util/src/test.rs rename to svc/pkg/cluster/src/util/test.rs diff --git a/svc/pkg/cluster/src/workflows/cluster.rs b/svc/pkg/cluster/src/workflows/cluster.rs new file mode 100644 index 000000000..a076da725 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/cluster.rs @@ -0,0 +1,156 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::types::{BuildDeliveryMethod, Pool, Provider}; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub cluster_id: Uuid, + pub name_id: String, + pub owner_team_id: Option, +} + +#[workflow] +pub async fn cluster(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + ctx.activity(InsertDbInput { + cluster_id: input.cluster_id, + name_id: input.name_id.clone(), + owner_team_id: input.owner_team_id, + }) + .await?; + + ctx.msg( + json!({ + "cluster_id": input.cluster_id, + }), + CreateComplete {}, + ) + .await?; + + let cluster_id = input.cluster_id; + loop { + match ctx.listen::
().await? { + Main::GameLink(sig) => { + ctx.activity(GameLinkInput { + cluster_id, + game_id: sig.game_id, + }) + .await?; + + ctx.msg( + json!({ + "cluster_id": cluster_id, + }), + GameLinkComplete {}, + ) + .await?; + } + Main::DatacenterCreate(sig) => { + ctx.dispatch_tagged_workflow( + &json!({ + "datacenter_id": sig.datacenter_id, + }), + crate::workflows::datacenter::Input { + cluster_id: input.cluster_id, + datacenter_id: sig.datacenter_id, + name_id: sig.name_id, + display_name: sig.display_name, + + provider: sig.provider, + provider_datacenter_id: sig.provider_datacenter_id, + provider_api_token: sig.provider_api_token, + + pools: sig.pools, + + build_delivery_method: sig.build_delivery_method, + prebakes_enabled: sig.prebakes_enabled, + }, + ) + .await?; + } + } + } +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct InsertDbInput { + cluster_id: Uuid, + name_id: String, + owner_team_id: Option, +} + +#[activity(InsertDb)] +async fn insert_db(ctx: &ActivityCtx, input: &InsertDbInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + INSERT INTO db_cluster.clusters ( + cluster_id, + name_id, + owner_team_id, + create_ts + ) + VALUES ($1, $2, $3, $4) + ", + input.cluster_id, + &input.name_id, + input.owner_team_id, + util::timestamp::now(), + ) + .await?; + + Ok(()) +} + +#[message("cluster-create-complete")] +pub struct CreateComplete {} + +#[signal("cluster-game-link")] +pub struct GameLink { + pub game_id: Uuid, +} + +#[signal("cluster-datacenter-create")] +pub struct DatacenterCreate { + pub datacenter_id: Uuid, + pub name_id: String, + pub display_name: String, + + pub provider: Provider, + pub provider_datacenter_id: String, + pub provider_api_token: Option, + + pub pools: Vec, + + pub build_delivery_method: BuildDeliveryMethod, + pub prebakes_enabled: bool, +} +join_signal!(Main, [GameLink, DatacenterCreate]); + +#[message("cluster-game-link-complete")] +pub struct GameLinkComplete {} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GameLinkInput { + cluster_id: Uuid, + game_id: Uuid, +} + +#[activity(GameLinkActivity)] +async fn game_link(ctx: &ActivityCtx, input: &GameLinkInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + INSERT INTO db_cluster.games ( + game_id, + cluster_id + ) + VALUES ($1, $2) + ", + input.game_id, + input.cluster_id, + ) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/src/workflows/datacenter/mod.rs b/svc/pkg/cluster/src/workflows/datacenter/mod.rs new file mode 100644 index 000000000..f02f6a1e5 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/datacenter/mod.rs @@ -0,0 +1,299 @@ +use chirp_workflow::prelude::*; +use futures_util::FutureExt; +use serde_json::json; + +pub mod scale; +pub mod tls_issue; + +use crate::types::{BuildDeliveryMethod, Pool, PoolType, PoolUpdate, Provider, TlsState}; + +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub cluster_id: Uuid, + pub datacenter_id: Uuid, + pub name_id: String, + pub display_name: String, + + pub provider: Provider, + pub provider_datacenter_id: String, + pub provider_api_token: Option, + + pub pools: Vec, + + pub build_delivery_method: BuildDeliveryMethod, + pub prebakes_enabled: bool, +} + +#[workflow] +pub(crate) async fn cluster_datacenter(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + ctx.activity(InsertDbInput { + cluster_id: input.cluster_id, + datacenter_id: input.datacenter_id, + name_id: input.name_id.clone(), + display_name: input.display_name.clone(), + + provider: input.provider.clone(), + provider_datacenter_id: input.provider_datacenter_id.clone(), + provider_api_token: input.provider_api_token.clone(), + + pools: input.pools.clone(), + + build_delivery_method: input.build_delivery_method.clone(), + prebakes_enabled: input.prebakes_enabled, + }) + .await?; + + // Wait for TLS issuing process + ctx.workflow(tls_issue::Input { + datacenter_id: input.datacenter_id, + renew: false, + }) + .await?; + + ctx.msg( + json!({ + "datacenter_id": input.datacenter_id, + }), + CreateComplete {}, + ) + .await?; + + // Scale + ctx.signal(ctx.workflow_id(), Scale {}).await?; + + let datacenter_id = input.datacenter_id; + loop { + match ctx.listen::
().await? { + Main::Update(sig) => { + ctx.activity(UpdateDbInput { + datacenter_id, + pools: sig.pools, + prebakes_enabled: sig.prebakes_enabled, + }) + .await?; + + // Scale + ctx.signal(ctx.workflow_id(), Scale {}).await?; + } + Main::Scale(_) => { + ctx.workflow(scale::Input { datacenter_id }).await?; + } + Main::ServerCreate(sig) => { + ctx.dispatch_tagged_workflow( + &json!({ + "server_id": sig.server_id, + }), + crate::workflows::server::Input { + datacenter_id, + server_id: sig.server_id, + pool_type: sig.pool_type, + provider: input.provider.clone(), + tags: sig.tags, + }, + ) + .await?; + } + Main::TlsRenew(_) => { + ctx.dispatch_workflow(tls_issue::Input { + datacenter_id, + renew: true, + }) + .await?; + } + } + } +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +struct InsertDbInput { + cluster_id: Uuid, + datacenter_id: Uuid, + name_id: String, + display_name: String, + + provider: Provider, + provider_datacenter_id: String, + provider_api_token: Option, + + pools: Vec, + + build_delivery_method: BuildDeliveryMethod, + prebakes_enabled: bool, +} + +#[activity(InsertDb)] +async fn insert_db(ctx: &ActivityCtx, input: &InsertDbInput) -> GlobalResult<()> { + let mut pools = input.pools.clone(); + + // Constrain the desired count + for pool in &mut pools { + pool.desired_count = pool.desired_count.max(pool.min_count).min(pool.max_count); + } + + let pools_buf = serde_json::to_string(&pools)?; + + rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { + let ctx = ctx.clone(); + let input = input.clone(); + let pools_buf = pools_buf.clone(); + + async move { + sql_execute!( + [ctx, @tx tx] + " + INSERT INTO db_cluster.datacenters ( + datacenter_id, + cluster_id, + name_id, + display_name, + provider2, + provider_datacenter_id, + provider_api_token, + pools2, + build_delivery_method2, + prebakes_enabled, + create_ts, + + -- Backwards compatibility + provider, + pools, + build_delivery_method + ) + VALUES ( + $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, + 0, b'', 0 + ) + ", + input.datacenter_id, + input.cluster_id, + &input.name_id, + &input.display_name, + serde_json::to_string(&input.provider)?, + &input.provider_datacenter_id, + &input.provider_api_token, + pools_buf, + serde_json::to_string(&input.build_delivery_method)?, + input.prebakes_enabled, + util::timestamp::now(), + ) + .await?; + + // Insert TLS record + sql_execute!( + [ctx, @tx tx] + " + INSERT INTO db_cluster.datacenter_tls ( + datacenter_id, + state2, + expire_ts, + + -- Backwards compatibility + state + ) + VALUES ($1, $2, 0, 0) + ", + input.datacenter_id, + serde_json::to_string(&TlsState::Creating)?, + ) + .await?; + + Ok(()) + } + .boxed() + }) + .await?; + + Ok(()) +} + +#[signal("cluster-datacenter-update")] +pub struct Update { + pub pools: Vec, + pub prebakes_enabled: Option, +} + +#[signal("cluster-datacenter-scale")] +pub struct Scale {} + +#[signal("cluster-datacenter-tls-renew")] +pub struct TlsRenew {} + +#[signal("cluster-datacenter-server-create")] +pub struct ServerCreate { + pub server_id: Uuid, + pub pool_type: PoolType, + pub tags: Vec, +} + +join_signal!(Main, [Update, Scale, ServerCreate, TlsRenew]); + +#[message("cluster-datacenter-create-complete")] +pub struct CreateComplete {} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UpdateDbInput { + datacenter_id: Uuid, + pools: Vec, + prebakes_enabled: Option, +} + +#[activity(UpdateDb)] +async fn update_db(ctx: &ActivityCtx, input: &UpdateDbInput) -> GlobalResult<()> { + // Get current pools + let (pools,) = sql_fetch_one!( + [ctx, (sqlx::types::Json>,)] + " + SELECT pools2 FROM db_cluster.datacenters + WHERE datacenter_id = $1 + ", + input.datacenter_id, + ) + .await?; + let mut pools = pools.0; + + for pool in &input.pools { + let current_pool = unwrap!( + pools.iter_mut().find(|p| p.pool_type == pool.pool_type), + "attempting to update pool that doesn't exist in current config" + ); + + // Update pool config + if !pool.hardware.is_empty() { + current_pool.hardware.clone_from(&pool.hardware); + } + if let Some(desired_count) = pool.desired_count { + current_pool.desired_count = desired_count; + } + if let Some(min_count) = pool.min_count { + current_pool.min_count = min_count; + } + if let Some(max_count) = pool.max_count { + current_pool.max_count = max_count; + } + if let Some(drain_timeout) = pool.drain_timeout { + current_pool.drain_timeout = drain_timeout; + } + } + + sql_execute!( + [ctx] + " + UPDATE db_cluster.datacenters + SET + pools = $2, + prebakes_enabled = coalesce($3, prebakes_enabled) + WHERE datacenter_id = $1 + ", + input.datacenter_id, + serde_json::to_string(&pools)?, + input.prebakes_enabled, + ) + .await?; + + // Purge cache + ctx.cache() + .purge("cluster.datacenters", [input.datacenter_id]) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/worker/src/workers/datacenter_scale.rs b/svc/pkg/cluster/src/workflows/datacenter/scale.rs similarity index 70% rename from svc/pkg/cluster/worker/src/workers/datacenter_scale.rs rename to svc/pkg/cluster/src/workflows/datacenter/scale.rs index 5a16b9484..4c6b89f11 100644 --- a/svc/pkg/cluster/worker/src/workers/datacenter_scale.rs +++ b/svc/pkg/cluster/src/workflows/datacenter/scale.rs @@ -1,4 +1,3 @@ -use std::convert::{TryFrom, TryInto}; // TERMINOLOGY: // // server: a non-destroyed non-tainted server @@ -8,24 +7,25 @@ use std::convert::{TryFrom, TryInto}; // draining server: a server that is currently draining, not drained // drained server: a server that is finished draining // tainted server: a tainted server + use std::{ cmp::Ordering, collections::HashMap, - future::Future, + convert::{TryFrom, TryInto}, iter::{DoubleEndedIterator, Iterator}, - pin::Pin, }; -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use futures_util::{FutureExt, StreamExt, TryStreamExt}; -use proto::backend::{self, pkg::*}; +use serde_json::json; -type MsgFuture = Pin> + Send>>; +use crate::types::{Datacenter, PoolType, Provider}; #[derive(sqlx::FromRow)] struct ServerRow { server_id: Uuid, pool_type: i64, + pool_type2: Option>, is_installed: bool, has_nomad_node: bool, is_draining: bool, @@ -35,7 +35,7 @@ struct ServerRow { struct Server { server_id: Uuid, - pool_type: backend::cluster::PoolType, + pool_type: PoolType, is_installed: bool, has_nomad_node: bool, drain_state: DrainState, @@ -48,7 +48,12 @@ impl TryFrom for Server { fn try_from(value: ServerRow) -> GlobalResult { Ok(Server { server_id: value.server_id, - pool_type: unwrap!(backend::cluster::PoolType::from_i32(value.pool_type as i32)), + // Handle backwards compatibility + pool_type: if let Some(pool_type) = value.pool_type2 { + pool_type.0 + } else { + value.pool_type.try_into()? + }, is_installed: value.is_installed, has_nomad_node: value.has_nomad_node, is_tainted: value.is_tainted, @@ -71,23 +76,134 @@ enum DrainState { struct PoolCtx { datacenter_id: Uuid, - provider: i32, - pool_type: backend::cluster::PoolType, + provider: Provider, + pool_type: PoolType, desired_count: usize, } -#[worker(name = "cluster-datacenter-scale")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub datacenter_id: Uuid, +} + +#[workflow] +pub async fn cluster_scale(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + let diff = ctx + .activity(CalculateDiffInput { + datacenter_id: input.datacenter_id, + }) + .await?; + + if !diff.actions.is_empty() { + tracing::info!(actions=?diff.actions, "dispatching signals"); + + futures_util::stream::iter( + diff.actions + .into_iter() + .map(|action| action.dispatch(ctx.step(), input.datacenter_id).boxed()), + ) + .buffer_unordered(16) + .try_collect::>() + .await?; + } + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CalculateDiffInput { + datacenter_id: Uuid, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CalculateDiffOutput { + actions: Vec, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +enum Action { + Provision { + server_id: Uuid, + pool_type: PoolType, + provider: Provider, + }, + Drain { + server_id: Uuid, + }, + Undrain { + server_id: Uuid, + }, + Destroy { + server_id: Uuid, + }, +} + +impl Action { + async fn dispatch(self, mut ctx: WorkflowCtx, datacenter_id: Uuid) -> GlobalResult<()> { + match self { + Action::Provision { + server_id, + pool_type, + provider, + } => { + ctx.dispatch_tagged_workflow( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Input { + datacenter_id, + server_id, + pool_type, + provider, + tags: Vec::new(), + }, + ) + .await?; + } + Action::Drain { server_id } => { + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Drain {}, + ) + .await?; + } + Action::Undrain { server_id } => { + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Undrain {}, + ) + .await?; + } + Action::Destroy { server_id } => { + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + crate::workflows::server::Destroy {}, + ) + .await?; + } + } + + Ok(()) + } +} + +#[activity(CalculateDiff)] +async fn calculate_diff( + ctx: &ActivityCtx, + input: &CalculateDiffInput, +) -> GlobalResult { let (datacenter_res, topology_res) = tokio::try_join!( - op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], + ctx.op(crate::ops::datacenter::get::Input { + datacenter_ids: vec![input.datacenter_id], }), - op!([ctx] cluster_datacenter_topology_get { - datacenter_ids: vec![datacenter_id.into()], + ctx.op(crate::ops::datacenter::topology_get::Input { + datacenter_ids: vec![input.datacenter_id], }), )?; @@ -97,17 +213,12 @@ async fn worker( let memory_by_server = topology .servers .iter() - .map(|server| { - Ok(( - unwrap_ref!(server.server_id).as_uuid(), - unwrap_ref!(server.usage).memory, - )) - }) + .map(|server| Ok((server.server_id, server.usage.memory))) .collect::>>()?; // Run everything in a locking transaction - let msgs = rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - let ctx = ctx.base(); + let actions = rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { + let ctx = ctx.clone(); let dc = dc.clone(); let memory_by_server = memory_by_server.clone(); @@ -115,32 +226,20 @@ async fn worker( }) .await?; - // Publish all messages - if !msgs.is_empty() { - tracing::info!("transaction successful, publishing messages"); - - futures_util::stream::iter(msgs) - .buffer_unordered(16) - .try_collect::>() - .await?; - } - - Ok(()) + Ok(CalculateDiffOutput { actions }) } async fn inner( - ctx: OperationContext<()>, + ctx: ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - dc: backend::cluster::Datacenter, + dc: Datacenter, memory_by_server: HashMap, -) -> GlobalResult> { - let datacenter_id = unwrap_ref!(dc.datacenter_id).as_uuid(); - +) -> GlobalResult> { let servers = sql_fetch_all!( [ctx, ServerRow, @tx tx] " SELECT - server_id, pool_type, + server_id, pool_type, pool_type2, (install_complete_ts IS NOT NULL) AS is_installed, (nomad_node_id IS NOT NULL) AS has_nomad_node, (drain_ts IS NOT NULL) AS is_draining, @@ -155,7 +254,7 @@ async fn inner( ORDER BY create_ts DESC FOR UPDATE ", - datacenter_id, + dc.datacenter_id, ) .await?; @@ -170,36 +269,36 @@ async fn inner( // TODO: RVT-3732 Sort gg and ats servers by cpu usage // servers.sort_by_key - let mut msgs = Vec::new(); + let mut actions = Vec::new(); // NOTE: Can't parallelize because this is in a transaction for pool in &dc.pools { let pool_ctx = PoolCtx { - datacenter_id, - provider: dc.provider, - pool_type: unwrap!(backend::cluster::PoolType::from_i32(pool.pool_type)), + datacenter_id: dc.datacenter_id, + provider: dc.provider.clone(), + pool_type: pool.pool_type.clone(), desired_count: pool.desired_count.max(pool.min_count).min(pool.max_count) as usize, }; - scale_servers(&ctx, tx, &mut msgs, &servers, &pool_ctx).await?; + scale_servers(&ctx, tx, &mut actions, &servers, &pool_ctx).await?; match pool_ctx.pool_type { - backend::cluster::PoolType::Job => { - cleanup_tainted_job_servers(&ctx, tx, &mut msgs, &servers, &pool_ctx).await? + PoolType::Job => { + cleanup_tainted_job_servers(&ctx, tx, &mut actions, &servers, &pool_ctx).await? } - _ => cleanup_tainted_servers(&ctx, tx, &mut msgs, &servers, &pool_ctx).await?, + _ => cleanup_tainted_servers(&ctx, tx, &mut actions, &servers, &pool_ctx).await?, } } - destroy_drained_servers(&ctx, tx, &mut msgs, &servers).await?; + destroy_drained_servers(&ctx, tx, &mut actions, &servers).await?; - Ok(msgs) + Ok(actions) } async fn scale_servers( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, servers: &[Server], pctx: &PoolCtx, ) -> GlobalResult<()> { @@ -217,21 +316,21 @@ async fn scale_servers( match pctx.desired_count.cmp(&active_count) { Ordering::Less => match pctx.pool_type { - backend::cluster::PoolType::Job => { - scale_down_job_servers(ctx, tx, msgs, pctx, active_servers_in_pool, active_count) + PoolType::Job => { + scale_down_job_servers(ctx, tx, actions, pctx, active_servers_in_pool, active_count) .await? } - backend::cluster::PoolType::Gg => { - scale_down_gg_servers(ctx, tx, msgs, pctx, active_servers_in_pool, active_count) + PoolType::Gg => { + scale_down_gg_servers(ctx, tx, actions, pctx, active_servers_in_pool, active_count) .await? } - backend::cluster::PoolType::Ats => { - scale_down_ats_servers(ctx, tx, msgs, pctx, active_servers_in_pool, active_count) + PoolType::Ats => { + scale_down_ats_servers(ctx, tx, actions, pctx, active_servers_in_pool, active_count) .await? } }, Ordering::Greater => { - scale_up_servers(ctx, tx, msgs, pctx, servers_in_pool, active_count).await?; + scale_up_servers(ctx, tx, actions, pctx, servers_in_pool, active_count).await?; } Ordering::Equal => {} } @@ -240,9 +339,9 @@ async fn scale_servers( } async fn scale_down_job_servers<'a, I: Iterator>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, pctx: &PoolCtx, active_servers: I, active_count: usize, @@ -269,7 +368,7 @@ async fn scale_down_job_servers<'a, I: Iterator>( .take(destroy_count) .map(|server| server.server_id); - destroy_servers(ctx, tx, msgs, destroy_candidates).await?; + destroy_servers(ctx, tx, actions, destroy_candidates).await?; } // Drain servers @@ -282,16 +381,16 @@ async fn scale_down_job_servers<'a, I: Iterator>( .take(drain_count) .map(|server| server.server_id); - drain_servers(ctx, tx, msgs, drain_candidates).await?; + drain_servers(ctx, tx, actions, drain_candidates).await?; } Ok(()) } async fn scale_down_gg_servers<'a, I: Iterator + DoubleEndedIterator + Clone>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, pctx: &PoolCtx, active_servers: I, active_count: usize, @@ -314,7 +413,7 @@ async fn scale_down_gg_servers<'a, I: Iterator + DoubleEndedI .take(drain_count) .map(|server| server.server_id); - drain_servers(ctx, tx, msgs, drain_candidates).await?; + drain_servers(ctx, tx, actions, drain_candidates).await?; } Ok(()) @@ -324,9 +423,9 @@ async fn scale_down_ats_servers< 'a, I: Iterator + DoubleEndedIterator + Clone, >( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, pctx: &PoolCtx, active_servers: I, active_count: usize, @@ -349,16 +448,16 @@ async fn scale_down_ats_servers< .take(drain_count) .map(|server| server.server_id); - drain_servers(ctx, tx, msgs, drain_candidates).await?; + drain_servers(ctx, tx, actions, drain_candidates).await?; } Ok(()) } async fn scale_up_servers<'a, I: Iterator + Clone>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, pctx: &PoolCtx, servers: I, active_count: usize, @@ -391,7 +490,7 @@ async fn scale_up_servers<'a, I: Iterator + Clone>( .take(undrain_count) .map(|server| server.server_id); - undrain_servers(ctx, tx, msgs, undrain_candidates).await?; + undrain_servers(ctx, tx, actions, undrain_candidates).await?; } // Create new servers @@ -399,7 +498,7 @@ async fn scale_up_servers<'a, I: Iterator + Clone>( tracing::info!(count=%provision_count, "provisioning servers"); for _ in 0..provision_count { - provision_server(ctx, tx, msgs, pctx).await?; + provision_server(ctx, tx, actions, pctx).await?; } } @@ -407,9 +506,9 @@ async fn scale_up_servers<'a, I: Iterator + Clone>( } async fn cleanup_tainted_job_servers( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, servers: &[Server], pctx: &PoolCtx, ) -> GlobalResult<()> { @@ -455,7 +554,7 @@ async fn cleanup_tainted_job_servers( destroy_servers( ctx, tx, - msgs, + actions, without_nomad_servers .iter() .take(destroy_count) @@ -477,7 +576,7 @@ async fn cleanup_tainted_job_servers( drain_servers( ctx, tx, - msgs, + actions, nomad_servers .iter() .take(drain_count) @@ -490,9 +589,9 @@ async fn cleanup_tainted_job_servers( } async fn cleanup_tainted_servers( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, servers: &[Server], pctx: &PoolCtx, ) -> GlobalResult<()> { @@ -532,7 +631,7 @@ async fn cleanup_tainted_servers( drain_servers( ctx, tx, - msgs, + actions, active_tainted_servers_in_pool .take(drain_count) .map(|server| server.server_id), @@ -545,9 +644,9 @@ async fn cleanup_tainted_servers( // Destroys all drained servers (including tainted drained servers) async fn destroy_drained_servers( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, servers: &[Server], ) -> GlobalResult<()> { let drained_server_ids = servers @@ -562,13 +661,13 @@ async fn destroy_drained_servers( tracing::info!(count=%drained_server_ids.len(), "destroying drained servers"); - destroy_servers(ctx, tx, msgs, drained_server_ids.into_iter()).await + destroy_servers(ctx, tx, actions, drained_server_ids.into_iter()).await } async fn drain_servers + Clone>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, server_ids: I, ) -> GlobalResult<()> { tracing::info!(count=%server_ids.clone().count(), "draining servers"); @@ -586,26 +685,15 @@ async fn drain_servers + Clone>( ) .await?; - msgs.extend(server_ids.map(|server_id| { - let ctx = ctx.base(); - async move { - tracing::info!(%server_id, "draining server"); - - msg!([ctx] cluster::msg::server_drain(server_id) { - server_id: Some(server_id.into()), - }) - .await - } - .boxed() - })); + actions.extend(server_ids.map(|server_id| Action::Drain { server_id })); Ok(()) } async fn undrain_servers + Clone>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, server_ids: I, ) -> GlobalResult<()> { tracing::info!(count=%server_ids.clone().count(), "undraining servers"); @@ -622,26 +710,15 @@ async fn undrain_servers + Clone>( ) .await?; - msgs.extend(server_ids.map(|server_id| { - let ctx = ctx.base(); - async move { - tracing::info!(%server_id, "undraining server"); - - msg!([ctx] cluster::msg::server_undrain(server_id) { - server_id: Some(server_id.into()), - }) - .await - } - .boxed() - })); + actions.extend(server_ids.map(|server_id| Action::Undrain { server_id })); Ok(()) } async fn provision_server( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, pctx: &PoolCtx, ) -> GlobalResult<()> { let server_id = Uuid::new_v4(); @@ -653,45 +730,34 @@ async fn provision_server( INSERT INTO db_cluster.servers ( server_id, datacenter_id, - pool_type, - create_ts + pool_type2, + create_ts, + + -- Backwards compatibility + pool_type ) - VALUES ($1, $2, $3, $4) + VALUES ($1, $2, $3, $4, 0) ", server_id, pctx.datacenter_id, - pctx.pool_type as i64, + serde_json::to_string(&pctx.pool_type)?, util::timestamp::now(), ) .await?; - let ctx = ctx.base(); - let datacenter_id = pctx.datacenter_id; - let provider = pctx.provider; - let pool_type = pctx.pool_type; - - msgs.push( - async move { - tracing::info!(%server_id, "provisioning server"); - msg!([ctx] cluster::msg::server_provision(server_id) { - datacenter_id: Some(datacenter_id.into()), - server_id: Some(server_id.into()), - pool_type: pool_type as i32, - provider: provider, - tags: Vec::new(), - }) - .await - } - .boxed(), - ); + actions.push(Action::Provision { + server_id, + pool_type: pctx.pool_type.clone(), + provider: pctx.provider.clone(), + }); Ok(()) } async fn destroy_servers + Clone>( - ctx: &OperationContext<()>, + ctx: &ActivityCtx, tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - msgs: &mut Vec, + actions: &mut Vec, server_ids: I, ) -> GlobalResult<()> { tracing::info!(count=%server_ids.clone().count(), "destroying servers"); @@ -709,19 +775,7 @@ async fn destroy_servers + Clone>( ) .await?; - msgs.extend(server_ids.map(|server_id| { - let ctx = ctx.base(); - async move { - tracing::info!(%server_id, "destroying server"); - - msg!([ctx] cluster::msg::server_destroy(server_id) { - server_id: Some(server_id.into()), - force: false, - }) - .await - } - .boxed() - })); + actions.extend(server_ids.map(|server_id| Action::Destroy { server_id })); Ok(()) } diff --git a/svc/pkg/cluster/worker/src/workers/datacenter_tls_issue.rs b/svc/pkg/cluster/src/workflows/datacenter/tls_issue.rs similarity index 70% rename from svc/pkg/cluster/worker/src/workers/datacenter_tls_issue.rs rename to svc/pkg/cluster/src/workflows/datacenter/tls_issue.rs index 91b28afa7..d2cb81854 100644 --- a/svc/pkg/cluster/worker/src/workers/datacenter_tls_issue.rs +++ b/svc/pkg/cluster/src/workflows/datacenter/tls_issue.rs @@ -1,12 +1,11 @@ use acme_lib::{ create_p384_key, persist::{MemoryPersist, Persist, PersistKey, PersistKind}, - Account, Certificate, Directory, DirectoryUrl, + Account, Directory, DirectoryUrl, }; -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use cloudflare::{endpoints as cf, framework as cf_framework, framework::async_api::ApiClient}; use futures_util::StreamExt; -use proto::backend::{self, pkg::*}; use tokio::task; use trust_dns_resolver::{ config::{ResolverConfig, ResolverOpts}, @@ -14,30 +13,22 @@ use trust_dns_resolver::{ TokioAsyncResolver, }; -use crate::util::CloudflareError; +use crate::{types::TlsState, util::cf_client}; const ENCRYPT_EMAIL: &str = "letsencrypt@rivet.gg"; -#[worker(name = "cluster-datacenter-tls-issue", timeout = 300)] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - tracing::warn!("temp disabled"); - return Ok(()); - - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - - // Create CF client - let cf_token = util::env::read_secret(&["cloudflare", "terraform", "auth_token"]).await?; - let client = cf_framework::async_api::Client::new( - cf_framework::auth::Credentials::UserAuthToken { token: cf_token }, - Default::default(), - cf_framework::Environment::Production, - ) - .map_err(CloudflareError::from)?; +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub datacenter_id: Uuid, + pub renew: bool, +} - // Fetch ACME account registration - let account = acme_account().await?; +#[workflow] +pub(crate) async fn cluster_datacenter_tls_issue( + ctx: &mut WorkflowCtx, + input: &Input, +) -> GlobalResult<()> { + let datacenter_id = input.datacenter_id; let base_zone_id = unwrap!( util::env::cloudflare::zone::main::id(), @@ -47,104 +38,69 @@ async fn worker( let domain_main = unwrap!(util::env::domain_main(), "dns not enabled"); let domain_job = unwrap!(util::env::domain_job(), "dns not enabled"); - // NOTE: We don't use try_join because these run in parallel, the dns record needs to be deleted for each - // order upon failure - let (gg_cert, job_cert) = tokio::join!( - order( - &client, - ctx.renew, - base_zone_id, - &account, - domain_main, - vec![format!("*.{datacenter_id}.{domain_main}")], - ), - order( - &client, - ctx.renew, - job_zone_id, - &account, - domain_job, - vec![ - format!("*.lobby.{datacenter_id}.{domain_job}"), - format!("*.{datacenter_id}.{domain_job}"), - ], - ), - ); - let (gg_cert, job_cert) = (gg_cert?, job_cert?); + let (gg_cert, job_cert) = ctx + .join(( + OrderInput { + renew: input.renew, + zone_id: base_zone_id.to_string(), + common_name: domain_main.to_string(), + subject_alternative_names: vec![format!("*.{datacenter_id}.{domain_main}")], + }, + OrderInput { + renew: input.renew, + zone_id: job_zone_id.to_string(), + common_name: domain_job.to_string(), + subject_alternative_names: vec![ + format!("*.lobby.{datacenter_id}.{domain_job}"), + format!("*.{datacenter_id}.{domain_job}"), + ], + }, + )) + .await?; - sql_execute!( - [ctx] - " - UPDATE db_cluster.datacenter_tls - SET - gg_cert_pem = $2, - gg_private_key_pem = $3, - job_cert_pem = $4, - job_private_key_pem = $5, - state = $6, - expire_ts = $7 - WHERE datacenter_id = $1 - ", - datacenter_id, - gg_cert.certificate(), - gg_cert.private_key(), - job_cert.certificate(), - job_cert.private_key(), - backend::cluster::TlsState::Active as i64, - util::timestamp::now() + util::duration::days(gg_cert.valid_days_left()), - ) + ctx.activity(InsertDbInput { + datacenter_id: input.datacenter_id, + gg_cert: gg_cert.cert, + gg_private_key: gg_cert.private_key, + job_cert: job_cert.cert, + job_private_key: job_cert.private_key, + expire_ts: gg_cert.expire_ts, + }) .await?; Ok(()) } -async fn acme_account() -> GlobalResult> { - let url = match util::env::var("TLS_ACME_DIRECTORY")?.as_str() { - "lets_encrypt" => DirectoryUrl::LetsEncrypt, - "lets_encrypt_staging" => DirectoryUrl::LetsEncryptStaging, - x => bail!(format!("unknown ACME directory: {x}")), - }; - - let persist = MemoryPersist::new(); +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +struct OrderInput { + renew: bool, + zone_id: String, + common_name: String, + subject_alternative_names: Vec, +} - // Write account private key (from terraform) to persistence - let pem_key = PersistKey::new( - ENCRYPT_EMAIL, - PersistKind::AccountPrivateKey, - "acme_account", - ); - let pem = util::env::var("TLS_ACME_ACCOUNT_PRIVATE_KEY_PEM")?; - persist.put(&pem_key, pem.as_bytes())?; +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +struct OrderOutput { + cert: String, + private_key: String, + expire_ts: i64, +} - // Get ACME account info - let acc = tokio::task::spawn_blocking(move || { - // Initialize ACME directory - let dir = Directory::from_url(persist, url)?; +#[activity(Order)] +#[timeout = 90] +async fn order(ctx: &ActivityCtx, input: &OrderInput) -> GlobalResult { + let client = cf_client().await?; - tracing::info!("fetching account"); - dir.account(ENCRYPT_EMAIL) - }) - .await??; + // Fetch ACME account registration + let account = acme_account().await?; - Ok(acc) -} - -// TODO: This function contains both blocking calls that cannot be shared between threads and async calls. -// Maybe theres a way to defer the blocking calls somehow -async fn order( - client: &cf_framework::async_api::Client, - renew: bool, - zone_id: &str, - account: &Account

, - common_name: &str, - subject_alternative_names: Vec, -) -> GlobalResult { - tracing::info!(cn=%common_name, "creating order"); + tracing::info!(cn=%input.common_name, "creating order"); let mut order = task::block_in_place(|| { account.new_order( - common_name, - &subject_alternative_names + &input.common_name, + &input + .subject_alternative_names .iter() .map(|s| s.as_str()) .collect::>(), @@ -153,11 +109,15 @@ async fn order( // When not renewing, if the ownership of the domain(s) have already been authorized in a previous order // we might be able to skip validation. The ACME API provider decides. - let order_csr = if let Some(order_csr) = renew.then(|| order.confirm_validations()).flatten() { + let order_csr = if let Some(order_csr) = + input.renew.then(|| order.confirm_validations()).flatten() + { order_csr } else { + let client = &client; + loop { - tracing::info!(%common_name, "fetching authorizations"); + tracing::info!(cn=%input.common_name, "fetching authorizations"); let auths = task::block_in_place(|| order.authorizations())?; // Run authorizations in parallel @@ -168,7 +128,7 @@ async fn order( let hostname = format!("_acme-challenge.{}", auth.api_auth().identifier.value); let dns_record_id = - create_dns_record(client, zone_id, &hostname, &proof).await?; + create_dns_record(client, &input.zone_id, &hostname, &proof).await?; let try_block = async { // Wait for DNS to propagate @@ -182,7 +142,7 @@ async fn order( .await; // Delete regardless of success of the above try block - delete_dns_record(client, zone_id, &dns_record_id).await?; + delete_dns_record(client, &input.zone_id, &dns_record_id).await?; try_block } @@ -214,7 +174,44 @@ async fn order( tracing::info!("order finalized"); - Ok(cert) + Ok(OrderOutput { + cert: cert.certificate().to_string(), + private_key: cert.private_key().to_string(), + expire_ts: util::timestamp::now() + util::duration::days(cert.valid_days_left()), + }) +} + +async fn acme_account() -> GlobalResult> { + let url = match util::env::var("TLS_ACME_DIRECTORY")?.as_str() { + "lets_encrypt" => DirectoryUrl::LetsEncrypt, + "lets_encrypt_staging" => DirectoryUrl::LetsEncryptStaging, + x => bail!(format!("unknown ACME directory: {x}")), + }; + + tracing::info!("fetching account from directory {:?}", url); + + let persist = MemoryPersist::new(); + + // Write account private key (from terraform) to persistence + let pem_key = PersistKey::new( + ENCRYPT_EMAIL, + PersistKind::AccountPrivateKey, + "acme_account", + ); + let pem = util::env::var("TLS_ACME_ACCOUNT_PRIVATE_KEY_PEM")?; + persist.put(&pem_key, pem.as_bytes())?; + + // Get ACME account info + let acc = tokio::task::spawn_blocking(move || { + // Initialize ACME directory + let dir = Directory::from_url(persist, url)?; + + tracing::info!("fetching account"); + dir.account(ENCRYPT_EMAIL) + }) + .await??; + + Ok(acc) } async fn create_dns_record( @@ -328,3 +325,41 @@ async fn poll_txt_dns(hostname: &str, content: &str) -> GlobalResult<()> { bail!("dns not resolved"); } + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +struct InsertDbInput { + datacenter_id: Uuid, + gg_cert: String, + gg_private_key: String, + job_cert: String, + job_private_key: String, + expire_ts: i64, +} + +#[activity(InsertDb)] +async fn insert_db(ctx: &ActivityCtx, input: &InsertDbInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + UPDATE db_cluster.datacenter_tls + SET + gg_cert_pem = $2, + gg_private_key_pem = $3, + job_cert_pem = $4, + job_private_key_pem = $5, + state2 = $6, + expire_ts = $7 + WHERE datacenter_id = $1 + ", + input.datacenter_id, + &input.gg_cert, + &input.gg_private_key, + &input.job_cert, + &input.job_private_key, + serde_json::to_string(&TlsState::Active)?, + input.expire_ts, + ) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/src/workflows/mod.rs b/svc/pkg/cluster/src/workflows/mod.rs new file mode 100644 index 000000000..e4ac007c4 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/mod.rs @@ -0,0 +1,4 @@ +pub mod cluster; +pub mod datacenter; +pub mod prebake; +pub mod server; diff --git a/svc/pkg/cluster/src/workflows/prebake.rs b/svc/pkg/cluster/src/workflows/prebake.rs new file mode 100644 index 000000000..feb5359c6 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/prebake.rs @@ -0,0 +1,214 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::{ + types::{PoolType, Provider}, + workflows::server::{GetDcInput, Linode}, +}; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub datacenter_id: Uuid, + pub provider: Provider, + pub pool_type: PoolType, + pub install_script_hash: String, + pub tags: Vec, +} + +#[workflow] +pub async fn cluster_prebake(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + let dc = ctx + .activity(GetDcInput { + datacenter_id: input.datacenter_id, + }) + .await?; + + let prebake_server_id = ctx.activity(GenerateServerIdInput {}).await?; + + let mut tags = input.tags.clone(); + tags.push("prebake".to_string()); + + match input.provider { + Provider::Linode => { + ctx.dispatch_tagged_workflow( + &json!({ + "server_id": prebake_server_id, + }), + linode::workflows::server::Input { + server_id: prebake_server_id, + provider_datacenter_id: dc.provider_datacenter_id.clone(), + custom_image: None, + api_token: dc.provider_api_token.clone(), + hardware: linode::util::consts::PREBAKE_HARDWARE.to_string(), + firewall_preset: match input.pool_type { + PoolType::Job => linode::types::FirewallPreset::Job, + PoolType::Gg => linode::types::FirewallPreset::Gg, + PoolType::Ats => linode::types::FirewallPreset::Ats, + }, + vlan_ip: None, + tags, + }, + ) + .await?; + + match ctx.listen::().await? { + Linode::ProvisionComplete(sig) => { + // Install server + ctx.workflow(crate::workflows::server::install::Input { + datacenter_id: input.datacenter_id, + server_id: None, + public_ip: sig.public_ip, + pool_type: input.pool_type.clone(), + initialize_immediately: false, + }) + .await?; + + // Create image + let workflow_id = ctx + .dispatch_tagged_workflow( + &json!({ + "linode_id": sig.linode_id, + }), + linode::workflows::image::Input { + prebake_server_id, + api_token: dc.provider_api_token.clone(), + linode_id: sig.linode_id, + boot_disk_id: sig.boot_disk_id, + }, + ) + .await?; + + // Wait for image creation + let image_create_res = ctx + .listen::() + .await?; + + // Write image id to db + ctx.activity(UpdateDbInput { + provider: input.provider.clone(), + datacenter_id: input.datacenter_id, + pool_type: input.pool_type.clone(), + install_script_hash: input.install_script_hash.clone(), + image_id: image_create_res.image_id, + }) + .await?; + + // Destroy linode server after the image is complete + ctx.tagged_signal( + &json!({ + "server_id": prebake_server_id, + }), + linode::workflows::server::Destroy {}, + ) + .await?; + + // Wait for image workflow to get cleaned up by linode-gc after the image expires + ctx.wait_for_workflow::(workflow_id) + .await?; + } + Linode::ProvisionFailed(sig) => { + tracing::error!( + err=%sig.err, + "failed to provision prebake server" + ); + } + } + } + } + + ctx.activity(SetDestroyedInput { + provider: input.provider.clone(), + datacenter_id: input.datacenter_id, + pool_type: input.pool_type.clone(), + install_script_hash: input.install_script_hash.clone(), + }) + .await?; + // UPDATE server_images2 + // SET destroy_ts = $1 + // WHERE = + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GenerateServerIdInput {} + +#[activity(GenerateServerId)] +async fn generate_server_id( + ctx: &ActivityCtx, + input: &GenerateServerIdInput, +) -> GlobalResult { + let prebake_server_id = Uuid::new_v4(); + + ctx.update_workflow_tags(&json!({ + "server_id": prebake_server_id, + })) + .await?; + + Ok(prebake_server_id) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UpdateDbInput { + datacenter_id: Uuid, + provider: Provider, + pool_type: PoolType, + install_script_hash: String, + image_id: String, +} + +#[activity(UpdateDb)] +async fn update_db(ctx: &ActivityCtx, input: &UpdateDbInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + UPDATE db_cluster.server_images2 + SET provider_image_id = $5 + WHERE + provider = $1 AND + install_hash = $2 AND + datacenter_id = $3 AND + pool_type = $4 + ", + serde_json::to_string(&input.provider)?, + &input.install_script_hash, + input.datacenter_id, + serde_json::to_string(&input.pool_type)?, + &input.image_id, + ) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct SetDestroyedInput { + datacenter_id: Uuid, + provider: Provider, + pool_type: PoolType, + install_script_hash: String, +} + +#[activity(SetDestroyed)] +async fn set_destroyed(ctx: &ActivityCtx, input: &SetDestroyedInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + UPDATE db_cluster.server_images2 + SET destroy_ts = $5 + WHERE + provider = $1 AND + install_hash = $2 AND + datacenter_id = $3 AND + pool_type = $4 + ", + serde_json::to_string(&input.provider)?, + &input.install_script_hash, + input.datacenter_id, + serde_json::to_string(&input.pool_type)?, + util::timestamp::now(), + ) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/src/workflows/server/dns_create.rs b/svc/pkg/cluster/src/workflows/server/dns_create.rs new file mode 100644 index 000000000..2da1bb48c --- /dev/null +++ b/svc/pkg/cluster/src/workflows/server/dns_create.rs @@ -0,0 +1,216 @@ +use std::net::{IpAddr, Ipv4Addr}; + +use chirp_workflow::prelude::*; +use cloudflare::{endpoints as cf, framework::async_api::ApiClient}; + +use crate::util::cf_client; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub server_id: Uuid, +} + +#[workflow] +pub async fn cluster_server_dns_create(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + ctx.activity(InsertDbInput { + server_id: input.server_id, + }) + .await?; + + let server_res = ctx + .activity(GetServerInfoInput { + server_id: input.server_id, + }) + .await?; + + let zone_id = unwrap!(util::env::cloudflare::zone::job::id(), "dns not configured"); + + ctx.activity(CreatePrimaryDnsRecordInput { + datacenter_id: server_res.datacenter_id, + server_id: input.server_id, + public_ip: server_res.public_ip, + zone_id: zone_id.to_string(), + }) + .await?; + + ctx.activity(CreateSecondaryDnsRecordInput { + datacenter_id: server_res.datacenter_id, + server_id: input.server_id, + public_ip: server_res.public_ip, + zone_id: zone_id.to_string(), + }) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct InsertDbInput { + server_id: Uuid, +} + +#[activity(InsertDb)] +async fn insert_db(ctx: &ActivityCtx, input: &InsertDbInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + INSERT INTO db_cluster.servers_cloudflare (server_id) + VALUES ($1) + ", + input.server_id, + ) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetServerInfoInput { + server_id: Uuid, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetServerInfoOutput { + datacenter_id: Uuid, + public_ip: Ipv4Addr, +} + +#[activity(GetServerInfo)] +async fn get_server_info( + ctx: &ActivityCtx, + input: &GetServerInfoInput, +) -> GlobalResult { + let (datacenter_id, public_ip) = sql_fetch_one!( + [ctx, (Uuid, IpAddr)] + " + SELECT datacenter_id, public_ip + FROM db_cluster.servers + WHERE server_id = $1 + ", + input.server_id, + ) + .await?; + + let public_ip = match public_ip { + IpAddr::V4(ip) => ip, + IpAddr::V6(_) => bail!("unexpected ipv6 public ip"), + }; + + Ok(GetServerInfoOutput { + datacenter_id, + public_ip, + }) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreatePrimaryDnsRecordInput { + datacenter_id: Uuid, + server_id: Uuid, + public_ip: Ipv4Addr, + zone_id: String, +} + +#[activity(CreatePrimaryDnsRecord)] +async fn create_primary_dns_record( + ctx: &ActivityCtx, + input: &CreatePrimaryDnsRecordInput, +) -> GlobalResult<()> { + let client = cf_client().await?; + + let record_name = format!( + "*.lobby.{}.{}", + input.datacenter_id, + unwrap!(util::env::domain_job()), + ); + let create_record_res = client + .request(&cf::dns::CreateDnsRecord { + zone_identifier: &input.zone_id, + params: cf::dns::CreateDnsRecordParams { + name: &record_name, + content: cf::dns::DnsContent::A { + content: input.public_ip, + }, + proxied: Some(false), + ttl: Some(60), + priority: None, + }, + }) + .await?; + + tracing::info!(record_id=%create_record_res.result.id, "created dns record"); + + // Save record id for deletion + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers_cloudflare + SET dns_record_id = $2 + WHERE + server_id = $1 AND + destroy_ts IS NULL + ", + input.server_id, + create_record_res.result.id, + ) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateSecondaryDnsRecordInput { + datacenter_id: Uuid, + server_id: Uuid, + public_ip: Ipv4Addr, + zone_id: String, +} + +// This is solely for compatibility with Discord activities. Their docs say they support parameter +// mapping but it does not work +// https://discord.com/developers/docs/activities/development-guides#prefixtarget-formatting-rules +#[activity(CreateSecondaryDnsRecord)] +async fn create_secondary_dns_record( + ctx: &ActivityCtx, + input: &CreateSecondaryDnsRecordInput, +) -> GlobalResult<()> { + let client = cf_client().await?; + + let secondary_record_name = format!( + "lobby.{}.{}", + input.datacenter_id, + unwrap!(util::env::domain_job()), + ); + let create_secondary_record_res = client + .request(&cf::dns::CreateDnsRecord { + zone_identifier: &input.zone_id, + params: cf::dns::CreateDnsRecordParams { + name: &secondary_record_name, + content: cf::dns::DnsContent::A { + content: input.public_ip, + }, + proxied: Some(false), + ttl: Some(60), + priority: None, + }, + }) + .await?; + + tracing::info!(record_id=%create_secondary_record_res.result.id, "created secondary dns record"); + + // Save record id for deletion + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers_cloudflare + SET secondary_dns_record_id = $2 + WHERE + server_id = $1 AND + destroy_ts IS NULL + ", + input.server_id, + create_secondary_record_res.result.id, + ) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/src/workflows/server/dns_delete.rs b/svc/pkg/cluster/src/workflows/server/dns_delete.rs new file mode 100644 index 000000000..24ec17329 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/server/dns_delete.rs @@ -0,0 +1,130 @@ +use chirp_workflow::prelude::*; +use cloudflare::{endpoints as cf, framework as cf_framework, framework::async_api::ApiClient}; + +use crate::util::cf_client; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub server_id: Uuid, +} + +#[workflow] +pub async fn cluster_server_dns_delete(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + let records_res = ctx + .activity(GetDnsRecordsInput { + server_id: input.server_id, + }) + .await?; + + let zone_id = unwrap!(util::env::cloudflare::zone::job::id(), "dns not configured"); + + if let Some(dns_record_id) = records_res.dns_record_id { + ctx.activity(DeleteDnsRecordInput { + dns_record_id, + zone_id: zone_id.to_string(), + }) + .await?; + } else { + tracing::warn!("server has no primary dns record"); + } + + if let Some(dns_record_id) = records_res.secondary_dns_record_id { + ctx.activity(DeleteDnsRecordInput { + dns_record_id, + zone_id: zone_id.to_string(), + }) + .await?; + } else { + tracing::warn!("server has no secondary dns record"); + } + + ctx.activity(UpdateDbInput { + server_id: input.server_id, + }) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetDnsRecordsInput { + server_id: Uuid, +} + +#[derive(Debug, sqlx::FromRow, Serialize, Deserialize, Hash)] +struct GetDnsRecordsOutput { + dns_record_id: Option, + secondary_dns_record_id: Option, +} + +#[activity(GetDnsRecords)] +async fn get_dns_Records( + ctx: &ActivityCtx, + input: &GetDnsRecordsInput, +) -> GlobalResult { + sql_fetch_one!( + [ctx, GetDnsRecordsOutput] + " + SELECT dns_record_id, secondary_dns_record_id + FROM db_cluster.servers_cloudflare + WHERE + server_id = $1 AND + destroy_ts IS NULL + ", + &input.server_id, + ) + .await +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct DeleteDnsRecordInput { + dns_record_id: String, + zone_id: String, +} + +#[activity(DeleteDnsRecord)] +async fn delete_dns_record(ctx: &ActivityCtx, input: &DeleteDnsRecordInput) -> GlobalResult<()> { + let client = cf_client().await?; + + let res = client + .request(&cf::dns::DeleteDnsRecord { + zone_identifier: &input.zone_id, + identifier: &input.dns_record_id, + }) + .await; + + if let Err(cf_framework::response::ApiFailure::Error(http::status::StatusCode::NOT_FOUND, _)) = + res + { + tracing::warn!(zone_id=%input.zone_id, record_id=%input.dns_record_id, "dns record not found"); + } else { + res?; + tracing::info!(record_id=%input.dns_record_id, "deleted dns record"); + } + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UpdateDbInput { + server_id: Uuid, +} + +#[activity(UpdateDb)] +async fn update_db(ctx: &ActivityCtx, input: &UpdateDbInput) -> GlobalResult<()> { + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers_cloudflare + SET destroy_ts = $2 + WHERE + server_id = $1 AND + destroy_ts IS NULL + ", + input.server_id, + util::timestamp::now(), + ) + .await?; + + Ok(()) +} diff --git a/svc/pkg/cluster/src/workflows/server/drain.rs b/svc/pkg/cluster/src/workflows/server/drain.rs new file mode 100644 index 000000000..cdaa9100f --- /dev/null +++ b/svc/pkg/cluster/src/workflows/server/drain.rs @@ -0,0 +1,119 @@ +use chirp_workflow::prelude::*; +use nomad_client::{ + apis::{configuration::Configuration, nodes_api}, + models, +}; +use rivet_operation::prelude::proto::backend::pkg::mm; +use serde_json::json; + +use crate::types::PoolType; + +lazy_static::lazy_static! { + static ref NOMAD_CONFIG: Configuration = nomad_util::new_config_from_env().unwrap(); +} + +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub datacenter_id: Uuid, + pub server_id: Uuid, + pub pool_type: PoolType, + pub drain_timeout: u64, +} + +#[workflow] +pub(crate) async fn cluster_server_drain(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + match input.pool_type { + PoolType::Job => { + let started_drain = ctx + .activity(DrainNodeInput { + datacenter_id: input.datacenter_id, + server_id: input.server_id, + drain_timeout: input.drain_timeout, + }) + .await?; + + if !started_drain { + ctx.tagged_signal( + &json!({ + "server_id": input.server_id, + }), + crate::workflows::server::NomadDrainComplete {}, + ) + .await?; + } + } + PoolType::Gg => { + ctx.tagged_signal( + &json!({ + "server_id": input.server_id, + }), + crate::workflows::server::DnsDelete {}, + ) + .await?; + } + PoolType::Ats => {} + } + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct DrainNodeInput { + datacenter_id: Uuid, + server_id: Uuid, + drain_timeout: u64, +} + +#[activity(DrainNode)] +async fn drain_node(ctx: &ActivityCtx, input: &DrainNodeInput) -> GlobalResult { + let (nomad_node_id,) = sql_fetch_one!( + [ctx, (Option,)] + " + SELECT nomad_node_id + FROM db_cluster.servers + WHERE server_id = $1 + ", + input.server_id, + ) + .await?; + + if let Some(nomad_node_id) = nomad_node_id { + // Drain complete message is caught by `cluster-nomad-node-drain-complete` + nodes_api::update_node_drain( + &NOMAD_CONFIG, + &nomad_node_id, + models::NodeUpdateDrainRequest { + drain_spec: Some(Box::new(models::DrainSpec { + // In nanoseconds. `drain_timeout` must be below 292 years for this to not overflow + deadline: Some((input.drain_timeout * 1000000) as i64), + ignore_system_jobs: None, + })), + mark_eligible: None, + meta: None, + node_id: Some(nomad_node_id.clone()), + }, + None, + None, + None, + None, + None, + None, + None, + None, + None, + ) + .await?; + + // Prevent new matchmaker requests to the node running on this server + msg!([ctx] mm::msg::nomad_node_closed_set(&nomad_node_id) { + datacenter_id: Some(input.datacenter_id.into()), + nomad_node_id: nomad_node_id.clone(), + is_closed: true, + }) + .await?; + + Ok(true) + } else { + Ok(false) + } +} diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/mod.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/mod.rs similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/mod.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/mod.rs diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/nomad.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/nomad.rs similarity index 95% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/nomad.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/nomad.rs index acaaf99b9..b6fb4ec17 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/nomad.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/nomad.rs @@ -1,4 +1,4 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; pub fn install() -> String { include_str!("../files/nomad_install.sh").to_string() diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/ok_server.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/ok_server.rs similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/ok_server.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/ok_server.rs diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/rivet.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/rivet.rs similarity index 97% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/rivet.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/rivet.rs index f6224703c..15659c1b9 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/rivet.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/rivet.rs @@ -1,4 +1,4 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use super::TUNNEL_API_INTERNAL_PORT; diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/s3.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/s3.rs similarity index 98% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/s3.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/s3.rs index 23e2547b3..74a746f30 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/s3.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/s3.rs @@ -1,4 +1,4 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use indoc::formatdoc; use s3_util::Provider; diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traefik.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traefik.rs similarity index 99% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traefik.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traefik.rs index ea48c099a..c2373373c 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traefik.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traefik.rs @@ -1,6 +1,6 @@ use std::collections::HashMap; -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use indoc::formatdoc; use super::{ diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traffic_server.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traffic_server.rs similarity index 96% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traffic_server.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traffic_server.rs index d39f08282..d0c2338f1 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/traffic_server.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/traffic_server.rs @@ -1,4 +1,4 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use include_dir::{include_dir, Dir}; use s3_util::Provider; @@ -47,7 +47,7 @@ pub async fn configure() -> GlobalResult { } static TRAFFIC_SERVER_CONFIG_DIR: Dir<'_> = include_dir!( - "$CARGO_MANIFEST_DIR/src/workers/server_install/install_scripts/files/traffic_server" + "$CARGO_MANIFEST_DIR/src/workflows/server/install/install_scripts/files/traffic_server" ); async fn config() -> GlobalResult> { diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/vector.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/vector.rs similarity index 87% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/vector.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/components/vector.rs index 67ec8a199..129c91082 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/components/vector.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/components/vector.rs @@ -1,7 +1,8 @@ use std::collections::HashMap; -use chirp_worker::prelude::*; -use proto::backend; +use chirp_workflow::prelude::*; + +use crate::types::PoolType; pub const TUNNEL_VECTOR_PORT: u16 = 5020; pub const TUNNEL_VECTOR_TCP_JSON_PORT: u16 = 5021; @@ -19,7 +20,7 @@ pub struct PrometheusTarget { pub scrape_interval: usize, } -pub fn configure(config: &Config, pool_type: backend::cluster::PoolType) -> String { +pub fn configure(config: &Config, pool_type: PoolType) -> String { let sources = config .prometheus_targets .keys() @@ -28,9 +29,9 @@ pub fn configure(config: &Config, pool_type: backend::cluster::PoolType) -> Stri .join(", "); let pool_type_str = match pool_type { - backend::cluster::PoolType::Job => "job", - backend::cluster::PoolType::Gg => "gg", - backend::cluster::PoolType::Ats => "ats", + PoolType::Job => "job", + PoolType::Gg => "gg", + PoolType::Ats => "ats", }; let mut config_str = formatdoc!( diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/cni_plugins.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/cni_plugins.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/cni_plugins.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/cni_plugins.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/docker.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/docker.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/docker.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/docker.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/node_exporter.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/node_exporter.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/node_exporter.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/node_exporter.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/nomad_configure.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/nomad_configure.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/nomad_configure.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/nomad_configure.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/nomad_install.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/nomad_install.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/nomad_install.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/nomad_install.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/ok_server.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/ok_server.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/ok_server.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/ok_server.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_create_hook.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_create_hook.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_create_hook.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_create_hook.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_fetch_info.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_fetch_info.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_fetch_info.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_fetch_info.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_fetch_tls.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_fetch_tls.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/rivet_fetch_tls.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/rivet_fetch_tls.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/sysctl.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/sysctl.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/sysctl.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/sysctl.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traefik.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traefik.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traefik.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traefik.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traefik_instance.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traefik_instance.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traefik_instance.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traefik_instance.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/cache.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/cache.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/cache.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/cache.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/hosting.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/hosting.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/hosting.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/hosting.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/ip_allow.yaml b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/ip_allow.yaml similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/ip_allow.yaml rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/ip_allow.yaml diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/logging.yaml b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/logging.yaml similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/logging.yaml rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/logging.yaml diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/parent.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/parent.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/parent.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/parent.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/plugin.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/plugin.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/plugin.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/plugin.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/records.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/records.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/records.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/records.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/sni.yaml b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/sni.yaml similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/sni.yaml rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/sni.yaml diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/socks.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/socks.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/socks.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/socks.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/splitdns.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/splitdns.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/splitdns.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/splitdns.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/ssl_multicert.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/ssl_multicert.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/ssl_multicert.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/ssl_multicert.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/strategies.yaml b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/strategies.yaml similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/strategies.yaml rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/strategies.yaml diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/strip_headers.lua b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/strip_headers.lua similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/strip_headers.lua rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/strip_headers.lua diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/trafficserver-release b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/trafficserver-release similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/trafficserver-release rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/trafficserver-release diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/volume.config b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/volume.config similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server/etc/volume.config rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server/etc/volume.config diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server_configure.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server_configure.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server_configure.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server_configure.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server_install.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server_install.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/traffic_server_install.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/traffic_server_install.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/vector_configure.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/vector_configure.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/vector_configure.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/vector_configure.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/vector_install.sh b/svc/pkg/cluster/src/workflows/server/install/install_scripts/files/vector_install.sh similarity index 100% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/files/vector_install.sh rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/files/vector_install.sh diff --git a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/mod.rs b/svc/pkg/cluster/src/workflows/server/install/install_scripts/mod.rs similarity index 88% rename from svc/pkg/cluster/worker/src/workers/server_install/install_scripts/mod.rs rename to svc/pkg/cluster/src/workflows/server/install/install_scripts/mod.rs index 57e8c5859..4987a6da3 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/install_scripts/mod.rs +++ b/svc/pkg/cluster/src/workflows/server/install/install_scripts/mod.rs @@ -1,7 +1,8 @@ use std::collections::HashMap; -use chirp_worker::prelude::*; -use proto::backend; +use chirp_workflow::prelude::*; + +use crate::types::PoolType; pub mod components; @@ -11,7 +12,7 @@ const GG_TRAEFIK_INSTANCE_NAME: &str = "game_guard"; // This script installs all of the software that doesn't need to know anything about the server running // it (doesn't need to know server id, datacenter id, vlan ip, etc) pub async fn gen_install( - pool_type: backend::cluster::PoolType, + pool_type: PoolType, initialize_immediately: bool, server_token: &str, datacenter_id: Uuid, @@ -28,7 +29,7 @@ pub async fn gen_install( // MARK: Specific pool components match pool_type { - backend::cluster::PoolType::Job => { + PoolType::Job => { script.push(components::docker::install()); script.push(components::lz4::install()); script.push(components::skopeo::install()); @@ -37,7 +38,7 @@ pub async fn gen_install( script.push(components::cni::plugins()); script.push(components::nomad::install()); } - backend::cluster::PoolType::Gg => { + PoolType::Gg => { script.push(components::rivet::fetch_tls( initialize_immediately, server_token, @@ -46,7 +47,7 @@ pub async fn gen_install( )?); script.push(components::ok_server::install(initialize_immediately)); } - backend::cluster::PoolType::Ats => { + PoolType::Ats => { script.push(components::docker::install()); script.push(components::traffic_server::install()); } @@ -72,10 +73,7 @@ pub async fn gen_hook(server_token: &str) -> GlobalResult { // This script is templated on the server itself after fetching server data from the Rivet API (see gen_hook). // After being templated, it is run. -pub async fn gen_initialize( - pool_type: backend::cluster::PoolType, - datacenter_id: Uuid, -) -> GlobalResult { +pub async fn gen_initialize(pool_type: PoolType, datacenter_id: Uuid) -> GlobalResult { let mut script = Vec::new(); let mut prometheus_targets = HashMap::new(); @@ -91,7 +89,7 @@ pub async fn gen_initialize( // MARK: Specific pool components match pool_type { - backend::cluster::PoolType::Job => { + PoolType::Job => { script.push(components::nomad::configure()); prometheus_targets.insert( @@ -102,7 +100,7 @@ pub async fn gen_initialize( }, ); } - backend::cluster::PoolType::Gg => { + PoolType::Gg => { script.push(components::traefik::instance( components::traefik::Instance { name: GG_TRAEFIK_INSTANCE_NAME.to_string(), @@ -112,7 +110,7 @@ pub async fn gen_initialize( }, )); } - backend::cluster::PoolType::Ats => { + PoolType::Ats => { script.push(components::traffic_server::configure().await?); } } diff --git a/svc/pkg/cluster/worker/src/workers/server_install/mod.rs b/svc/pkg/cluster/src/workflows/server/install/mod.rs similarity index 52% rename from svc/pkg/cluster/worker/src/workers/server_install/mod.rs rename to svc/pkg/cluster/src/workflows/server/install/mod.rs index 4f4701455..d22b50643 100644 --- a/svc/pkg/cluster/worker/src/workers/server_install/mod.rs +++ b/svc/pkg/cluster/src/workflows/server/install/mod.rs @@ -1,57 +1,67 @@ use std::{ io::{Read, Write}, - net::TcpStream, + net::{Ipv4Addr, TcpStream}, path::Path, }; -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; +use rivet_operation::prelude::proto::{self, backend::pkg::token}; use ssh2::Session; -use util_cluster::metrics; + +use crate::{ + types::{Datacenter, PoolType}, + util::metrics, +}; mod install_scripts; -#[worker(name = "cluster-server-install", timeout = 200)] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - let datacenter_id = unwrap!(ctx.datacenter_id).as_uuid(); +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub datacenter_id: Uuid, + pub server_id: Option, + pub public_ip: Ipv4Addr, + pub pool_type: PoolType, + pub initialize_immediately: bool, +} - // Check for stale message - if ctx.req_dt() > util::duration::hours(1) { - tracing::warn!("discarding stale message"); +#[workflow] +pub(crate) async fn cluster_server_install( + ctx: &mut WorkflowCtx, + input: &Input, +) -> GlobalResult<()> { + let server_token = ctx.activity(CreateTokenInput {}).await?; - return Ok(()); - } + ctx.activity(InstallOverSshInput { + datacenter_id: input.datacenter_id, + public_ip: input.public_ip, + pool_type: input.pool_type.clone(), + initialize_immediately: input.initialize_immediately, + server_token, + }) + .await?; - if let Some(server_id) = ctx.server_id { - let (is_destroying_or_draining,) = sql_fetch_one!( - [ctx, (bool,)] - " - SELECT EXISTS( - SELECT 1 - FROM db_cluster.servers - WHERE server_id = $1 AND - (cloud_destroy_ts IS NOT NULL OR drain_ts IS NOT NULL) - ) - ", - server_id.as_uuid(), - ) + // If the server id is set this is not a prebake server + if let Some(server_id) = input.server_id { + ctx.activity(UpdateDbInput { + datacenter_id: input.datacenter_id, + server_id, + pool_type: input.pool_type.clone(), + }) .await?; - - if is_destroying_or_draining { - tracing::info!("server marked for deletion/drain, not installing"); - return Ok(()); - } } - let public_ip = ctx.public_ip.clone(); - let pool_type = unwrap!(backend::cluster::PoolType::from_i32(ctx.pool_type)); - let private_key_openssh = - util::env::read_secret(&["ssh", "server", "private_key_openssh"]).await?; + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateTokenInput {} +#[activity(CreateToken)] +async fn create_token(ctx: &ActivityCtx, input: &CreateTokenInput) -> GlobalResult { // Create server token for authenticating API calls from the server let token_res = op!([ctx] token_create { token_config: Some(token::create::request::TokenConfig { - ttl: util_cluster::SERVER_TOKEN_TTL, + ttl: crate::util::SERVER_TOKEN_TTL, }), refresh_token_config: None, issuer: "cluster-worker-server-install".to_owned(), @@ -71,22 +81,42 @@ async fn worker(ctx: &OperationContext) - ..Default::default() }) .await?; - let server_token = &unwrap_ref!(token_res.token).token; + let server_token = unwrap_ref!(token_res.token).token.clone(); + + Ok(server_token) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct InstallOverSshInput { + datacenter_id: Uuid, + public_ip: Ipv4Addr, + pool_type: PoolType, + initialize_immediately: bool, + server_token: String, +} + +#[activity(InstallOverSsh)] +#[timeout = 120] +async fn install_over_ssh(ctx: &ActivityCtx, input: &InstallOverSshInput) -> GlobalResult<()> { + let public_ip = input.public_ip; + let private_key_openssh = + util::env::read_secret(&["ssh", "server", "private_key_openssh"]).await?; let install_script = install_scripts::gen_install( - pool_type, - ctx.initialize_immediately, - server_token, - datacenter_id, + input.pool_type.clone(), + input.initialize_immediately, + &input.server_token, + input.datacenter_id, ) .await?; - let hook_script = install_scripts::gen_hook(server_token).await?; - let initialize_script = install_scripts::gen_initialize(pool_type, datacenter_id).await?; + let hook_script = install_scripts::gen_hook(&input.server_token).await?; + let initialize_script = + install_scripts::gen_initialize(input.pool_type.clone(), input.datacenter_id).await?; // Spawn blocking thread for ssh (no async support) tokio::task::spawn_blocking(move || { tracing::info!(%public_ip, "connecting over ssh"); - let tcp = TcpStream::connect((public_ip.as_str(), 22))?; + let tcp = TcpStream::connect((public_ip, 22))?; let mut sess = Session::new()?; sess.set_tcp_stream(tcp); @@ -130,7 +160,7 @@ async fn worker(ctx: &OperationContext) - channel.wait_close()?; if channel.exit_status()? != 0 { - tracing::error!(%stdout, %stderr, "failed to run script"); + tracing::error!(%public_ip, %stdout, %stderr, "failed to run script"); bail!("failed to run script"); } @@ -140,68 +170,6 @@ async fn worker(ctx: &OperationContext) - }) .await??; - let request_id = unwrap_ref!(ctx.request_id).as_uuid(); - msg!([ctx] cluster::msg::server_install_complete(request_id) { - request_id: ctx.request_id, - public_ip: ctx.public_ip.clone(), - datacenter_id: ctx.datacenter_id, - server_id: ctx.server_id, - provider: ctx.provider, - }) - .await?; - - // If the server id is set this is not a prebake server - if let Some(server_id) = ctx.server_id { - let install_complete_ts = util::timestamp::now(); - - let (provision_complete_ts,) = sql_fetch_one!( - [ctx, (i64,)] - " - UPDATE db_cluster.servers - SET install_complete_ts = $2 - WHERE server_id = $1 - RETURNING provision_complete_ts - ", - server_id.as_uuid(), - install_complete_ts, - ) - .await?; - - // Scale to get rid of tainted servers - msg!([ctx] @recursive cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: ctx.datacenter_id, - }) - .await?; - - // Create DNS record - if let backend::cluster::PoolType::Gg = pool_type { - // Source of truth record - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.servers_cloudflare (server_id) - VALUES ($1) - ", - server_id.as_uuid(), - ) - .await?; - - msg!([ctx] cluster::msg::server_dns_create(server_id) { - server_id: Some(server_id), - }) - .await?; - } - - insert_metrics( - ctx, - &pool_type, - datacenter_id, - install_complete_ts, - provision_complete_ts, - ) - .await?; - } - Ok(()) } @@ -238,36 +206,66 @@ fn write_script(sess: &Session, script_name: &str, content: &str) -> GlobalResul Ok(()) } -async fn insert_metrics( - ctx: &OperationContext, - pool_type: &backend::cluster::PoolType, +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UpdateDbInput { datacenter_id: Uuid, - install_complete_ts: i64, - provision_complete_ts: i64, -) -> GlobalResult<()> { - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) + server_id: Uuid, + pool_type: PoolType, +} + +#[activity(UpdateDb)] +async fn update_db(ctx: &ActivityCtx, input: &UpdateDbInput) -> GlobalResult<()> { + let install_complete_ts = util::timestamp::now(); + + let (provision_complete_ts,) = sql_fetch_one!( + [ctx, (i64,)] + " + UPDATE db_cluster.servers + SET install_complete_ts = $2 + WHERE server_id = $1 + RETURNING provision_complete_ts + ", + input.server_id, + install_complete_ts, + ) .await?; + + let datacenters_res = ctx + .op(crate::ops::datacenter::get::Input { + datacenter_ids: vec![input.datacenter_id], + }) + .await?; let dc = unwrap!(datacenters_res.datacenters.first()); - let datacenter_id = unwrap_ref!(dc.datacenter_id).as_uuid().to_string(); - let cluster_id = unwrap_ref!(dc.cluster_id).as_uuid().to_string(); + insert_metrics( + &input.pool_type, + dc, + install_complete_ts, + provision_complete_ts, + ); + + Ok(()) +} + +fn insert_metrics( + pool_type: &PoolType, + dc: &Datacenter, + install_complete_ts: i64, + provision_complete_ts: i64, +) { let dt = (install_complete_ts - provision_complete_ts) as f64 / 1000.0; metrics::INSTALL_DURATION .with_label_values(&[ - cluster_id.as_str(), - datacenter_id.as_str(), + &dc.cluster_id.to_string(), + &dc.datacenter_id.to_string(), &dc.provider_datacenter_id, &dc.name_id, match pool_type { - backend::cluster::PoolType::Job => "job", - backend::cluster::PoolType::Gg => "gg", - backend::cluster::PoolType::Ats => "ats", + PoolType::Job => "job", + PoolType::Gg => "gg", + PoolType::Ats => "ats", }, ]) .observe(dt); - - Ok(()) } diff --git a/svc/pkg/cluster/src/workflows/server/mod.rs b/svc/pkg/cluster/src/workflows/server/mod.rs new file mode 100644 index 000000000..4475a9b34 --- /dev/null +++ b/svc/pkg/cluster/src/workflows/server/mod.rs @@ -0,0 +1,796 @@ +use std::{ + convert::TryInto, + net::{IpAddr, Ipv4Addr}, +}; + +use chirp_workflow::prelude::*; +use rand::Rng; +use serde_json::json; + +pub(crate) mod dns_create; +pub(crate) mod dns_delete; +pub(crate) mod drain; +pub(crate) mod install; +pub(crate) mod undrain; + +use crate::{ + types::{Datacenter, PoolType, Provider}, + util::metrics, +}; + +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub datacenter_id: Uuid, + pub server_id: Uuid, + pub provider: Provider, + pub pool_type: PoolType, + pub tags: Vec, +} + +#[workflow] +pub(crate) async fn cluster_server(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + let dc = ctx + .activity(GetDcInput { + datacenter_id: input.datacenter_id, + }) + .await?; + + let pool = unwrap!( + dc.pools.iter().find(|p| p.pool_type == input.pool_type), + "datacenter does not have this type of pool configured" + ); + + // Get a new vlan ip + let vlan_ip = ctx + .activity(GetVlanIpInput { + datacenter_id: input.datacenter_id, + server_id: input.server_id, + pool_type: input.pool_type.clone(), + }) + .await?; + + let custom_image = if dc.prebakes_enabled { + let image_res = ctx + .activity(GetPrebakeInput { + datacenter_id: input.datacenter_id, + pool_type: input.pool_type.clone(), + provider: input.provider.clone(), + }) + .await?; + + // Start custom image creation process + if image_res.updated { + ctx.dispatch_workflow(crate::workflows::prebake::Input { + datacenter_id: input.datacenter_id, + provider: input.provider.clone(), + pool_type: input.pool_type.clone(), + install_script_hash: crate::util::INSTALL_SCRIPT_HASH.to_string(), + tags: Vec::new(), + }) + .await?; + } + + image_res.custom_image + } else { + None + }; + let already_installed = custom_image.is_some(); + + // Iterate through list of hardware and attempt to schedule a server. Goes to the next + // hardware if an error happens during provisioning + let mut hardware_list = pool.hardware.iter(); + let provision_res = loop { + // List exhausted + let Some(hardware) = hardware_list.next() else { + break None; + }; + + tracing::info!( + "attempting to provision hardware: {}", + hardware.provider_hardware, + ); + + match input.provider { + Provider::Linode => { + let workflow_id = ctx + .dispatch_tagged_workflow( + &json!({ + "server_id": input.server_id, + }), + linode::workflows::server::Input { + server_id: input.server_id, + provider_datacenter_id: dc.provider_datacenter_id.clone(), + custom_image: custom_image.clone(), + api_token: dc.provider_api_token.clone(), + hardware: hardware.provider_hardware.clone(), + firewall_preset: match input.pool_type { + PoolType::Job => linode::types::FirewallPreset::Job, + PoolType::Gg => linode::types::FirewallPreset::Gg, + PoolType::Ats => linode::types::FirewallPreset::Ats, + }, + vlan_ip: Some(vlan_ip), + tags: input.tags.clone(), + }, + ) + .await?; + + match ctx.listen::().await? { + Linode::ProvisionComplete(sig) => { + break Some(ProvisionResponse { + provider_server_workflow_id: workflow_id, + provider_server_id: sig.linode_id.to_string(), + provider_hardware: hardware.provider_hardware.clone(), + public_ip: sig.public_ip, + }); + } + Linode::ProvisionFailed(sig) => { + tracing::error!( + err=%sig.err, + server_id=?input.server_id, + "failed to provision server" + ); + } + } + } + } + }; + + let provider_server_workflow_id = if let Some(provision_res) = provision_res { + let provider_server_workflow_id = provision_res.provider_server_workflow_id; + let public_ip = provision_res.public_ip; + + ctx.activity(UpdateDbInput { + server_id: input.server_id, + pool_type: input.pool_type.clone(), + cluster_id: dc.cluster_id, + datacenter_id: dc.datacenter_id, + provider_datacenter_id: dc.provider_datacenter_id.clone(), + datacenter_name_id: dc.name_id.clone(), + provision_res, + already_installed, + }) + .await?; + + // Install components on server + if !already_installed { + ctx.workflow(install::Input { + datacenter_id: input.datacenter_id, + server_id: Some(input.server_id), + public_ip, + pool_type: input.pool_type.clone(), + initialize_immediately: true, + }) + .await?; + } + + // Scale to get rid of tainted servers + ctx.tagged_signal( + &json!({ + "datacenter_id": input.datacenter_id, + }), + crate::workflows::datacenter::Scale {}, + ) + .await?; + + // Create DNS record because the server is already installed + if let PoolType::Gg = input.pool_type { + ctx.workflow(dns_create::Input { + server_id: input.server_id, + }) + .await?; + } + + provider_server_workflow_id + } else { + tracing::error!( + server_id=?input.server_id, + hardware_options=?pool.hardware.len(), + "failed all attempts to provision server" + ); + + // Mark as destroyed (cleanup already occurred in the linode server workflow) + ctx.activity(MarkDestroyedInput { + server_id: input.server_id, + }) + .await?; + + bail!("failed all attempts to provision server"); + }; + + let mut state = State::default(); + loop { + match state.listen(ctx).await? { + Main::DnsCreate(_) => { + ctx.workflow(dns_create::Input { + server_id: input.server_id, + }) + .await?; + } + Main::DnsDelete(_) => { + ctx.workflow(dns_delete::Input { + server_id: input.server_id, + }) + .await?; + } + Main::NomadRegistered(sig) => { + ctx.activity(SetNomadNodeIdInput { + server_id: input.server_id, + cluster_id: dc.cluster_id, + datacenter_id: dc.datacenter_id, + provider_datacenter_id: dc.provider_datacenter_id.clone(), + datacenter_name_id: dc.name_id.clone(), + node_id: sig.node_id, + }) + .await?; + + // Scale to get rid of tainted servers + ctx.tagged_signal( + &json!({ + "datacenter_id": input.datacenter_id, + }), + crate::workflows::datacenter::Scale {}, + ) + .await?; + } + Main::NomadDrainComplete(_) => { + ctx.activity(SetDrainCompleteInput { + server_id: input.server_id, + }) + .await?; + + // Scale + ctx.tagged_signal( + &json!({ + "datacenter_id": input.datacenter_id, + }), + crate::workflows::datacenter::Scale {}, + ) + .await?; + } + Main::Drain(_) => { + ctx.workflow(drain::Input { + datacenter_id: input.datacenter_id, + server_id: input.server_id, + pool_type: input.pool_type.clone(), + drain_timeout: pool.drain_timeout, + }) + .await?; + } + Main::Undrain(_) => { + ctx.workflow(undrain::Input { + datacenter_id: input.datacenter_id, + server_id: input.server_id, + pool_type: input.pool_type.clone(), + }) + .await?; + } + Main::Taint(_) => {} // Only for state + Main::Destroy(_) => { + ctx.workflow(dns_delete::Input { + server_id: input.server_id, + }) + .await?; + + match input.provider { + Provider::Linode => { + tracing::info!(server_id=?input.server_id, "destroying linode server"); + + ctx.tagged_signal( + &json!({ + "server_id": input.server_id, + }), + linode::workflows::server::Destroy {}, + ) + .await?; + + // Wait for workflow to complete + ctx.wait_for_workflow::( + provider_server_workflow_id, + ) + .await?; + } + } + + break; + } + } + } + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +pub(crate) struct GetDcInput { + pub datacenter_id: Uuid, +} + +#[activity(GetDc)] +pub(crate) async fn get_dc(ctx: &ActivityCtx, input: &GetDcInput) -> GlobalResult { + let dcs_res = ctx + .op(crate::ops::datacenter::get::Input { + datacenter_ids: vec![input.datacenter_id], + }) + .await?; + let dc = unwrap!(dcs_res.datacenters.into_iter().next()); + + Ok(dc) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetVlanIpInput { + datacenter_id: Uuid, + server_id: Uuid, + pool_type: PoolType, +} + +#[activity(GetVlanIp)] +async fn get_vlan_ip(ctx: &ActivityCtx, input: &GetVlanIpInput) -> GlobalResult { + // Find next available vlan index + let mut vlan_addr_range = match input.pool_type { + PoolType::Job => util::net::job::vlan_addr_range(), + PoolType::Gg => util::net::gg::vlan_addr_range(), + PoolType::Ats => util::net::ats::vlan_addr_range(), + }; + let max_idx = vlan_addr_range.count() as i64; + + let (network_idx,) = sql_fetch_one!( + [ctx, (i64,)] + " + WITH + get_next_network_idx AS ( + SELECT mod(idx + $1, $2) AS idx + FROM generate_series(0, $2) AS s(idx) + WHERE NOT EXISTS ( + SELECT 1 + FROM db_cluster.servers + WHERE + pool_type2 = $3 AND + -- Technically this should check all servers where their datacenter's provider and + -- provider_datacenter_id are the same because VLAN is separated by irl datacenter + -- but this is good enough + datacenter_id = $4 AND + network_idx = mod(idx + $1, $2) AND + cloud_destroy_ts IS NULL + ) + LIMIT 1 + ), + update_network_idx AS ( + UPDATE db_cluster.servers + SET network_idx = (SELECT idx FROM get_next_network_idx) + WHERE server_id = $5 + RETURNING 1 + ) + SELECT idx FROM get_next_network_idx + ", + // Choose a random index to start from for better index spread + rand::thread_rng().gen_range(0i64..max_idx), + max_idx, + serde_json::to_string(&input.pool_type)?, + input.datacenter_id, + input.server_id, + ) + .await?; + + let vlan_ip = unwrap!(vlan_addr_range.nth(network_idx.try_into()?)); + + // Write vlan ip + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers + SET vlan_ip = $2 + WHERE server_id = $1 + ", + input.server_id, + IpAddr::V4(vlan_ip), + ) + .await?; + + Ok(vlan_ip) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetPrebakeInput { + datacenter_id: Uuid, + pool_type: PoolType, + provider: Provider, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetPrebakeOutput { + custom_image: Option, + updated: bool, +} + +#[activity(GetPrebake)] +async fn get_prebake(ctx: &ActivityCtx, input: &GetPrebakeInput) -> GlobalResult { + // Get the custom image id for this server, or insert a record and start creating one + let (image_id, updated) = sql_fetch_one!( + [ctx, (Option, bool)] + " + WITH + updated AS ( + INSERT INTO db_cluster.server_images2 AS s ( + provider, install_hash, datacenter_id, pool_type, create_ts + ) + VALUES ($1, $2, $3, $4, $5) + ON CONFLICT (provider, install_hash, datacenter_id, pool_type) DO UPDATE + SET + provider_image_id = NULL, + create_ts = $5 + WHERE s.create_ts < $6 + RETURNING provider, install_hash, datacenter_id, pool_type + ), + selected AS ( + SELECT provider, install_hash, datacenter_id, pool_type, provider_image_id + FROM db_cluster.server_images2 + WHERE + provider = $1 AND + install_hash = $2 AND + datacenter_id = $3 AND + pool_type = $4 + ) + SELECT + selected.provider_image_id, + -- Primary key is not null + (updated.provider IS NOT NULL) AS updated + FROM selected + FULL OUTER JOIN updated + ON true + ", + serde_json::to_string(&input.provider)?, + crate::util::INSTALL_SCRIPT_HASH, + input.datacenter_id, + serde_json::to_string(&input.pool_type)?, + util::timestamp::now(), + // 5 month expiration + util::timestamp::now() - util::duration::days(5 * 30), + ) + .await?; + + // Updated is true if this specific sql call either reset (if expired) or inserted the row + Ok(GetPrebakeOutput { + custom_image: if updated { None } else { image_id }, + updated, + }) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct ProvisionResponse { + provider_server_workflow_id: Uuid, + provider_server_id: String, + provider_hardware: String, + public_ip: Ipv4Addr, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UpdateDbInput { + server_id: Uuid, + pool_type: PoolType, + cluster_id: Uuid, + datacenter_id: Uuid, + provider_datacenter_id: String, + datacenter_name_id: String, + provision_res: ProvisionResponse, + already_installed: bool, +} + +#[activity(UpdateDb)] +async fn update_db(ctx: &ActivityCtx, input: &UpdateDbInput) -> GlobalResult<()> { + let provision_complete_ts = util::timestamp::now(); + + let (create_ts,) = sql_fetch_one!( + [ctx, (i64,)] + " + UPDATE db_cluster.servers + SET + provider_server_id = $2, + provider_hardware = $3, + public_ip = $4, + provision_complete_ts = $5, + install_complete_ts = $6 + WHERE server_id = $1 + RETURNING create_ts + ", + input.server_id, + &input.provision_res.provider_server_id, + &input.provision_res.provider_hardware, + IpAddr::V4(input.provision_res.public_ip), + provision_complete_ts, + if input.already_installed { + Some(provision_complete_ts) + } else { + None + }, + ) + .await?; + + insert_metrics( + input.cluster_id, + input.datacenter_id, + &input.provider_datacenter_id, + &input.datacenter_name_id, + &input.pool_type, + provision_complete_ts, + create_ts, + ) + .await; + + Ok(()) +} + +async fn insert_metrics( + cluster_id: Uuid, + datacenter_id: Uuid, + provider_datacenter_id: &str, + datacenter_name_id: &str, + pool_type: &PoolType, + provision_complete_ts: i64, + create_ts: i64, +) { + let dt = (provision_complete_ts - create_ts) as f64 / 1000.0; + + metrics::PROVISION_DURATION + .with_label_values(&[ + &cluster_id.to_string(), + &datacenter_id.to_string(), + provider_datacenter_id, + datacenter_name_id, + match pool_type { + PoolType::Job => "job", + PoolType::Gg => "gg", + PoolType::Ats => "ats", + }, + ]) + .observe(dt); +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct MarkDestroyedInput { + server_id: Uuid, +} + +#[activity(MarkDestroyed)] +async fn mark_destroyed(ctx: &ActivityCtx, input: &MarkDestroyedInput) -> GlobalResult<()> { + // Mark servers for destruction in db + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers + SET cloud_destroy_ts = $2 + WHERE server_id = $1 + ", + input.server_id, + util::timestamp::now(), + ) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct SetNomadNodeIdInput { + server_id: Uuid, + cluster_id: Uuid, + datacenter_id: Uuid, + provider_datacenter_id: String, + datacenter_name_id: String, + node_id: String, +} + +#[activity(SetNomadNodeId)] +async fn set_nomad_node_id(ctx: &ActivityCtx, input: &SetNomadNodeIdInput) -> GlobalResult<()> { + let nomad_join_ts = util::timestamp::now(); + + let (old_nomad_node_id, install_complete_ts) = sql_fetch_one!( + [ctx, (Option, Option)] + " + UPDATE db_cluster.servers + SET + nomad_node_id = $2, + nomad_join_ts = $3 + WHERE + server_id = $1 + RETURNING nomad_node_id, install_complete_ts + ", + input.server_id, + &input.node_id, + nomad_join_ts, + ) + .await?; + + if let Some(old_nomad_node_id) = old_nomad_node_id { + tracing::warn!(%old_nomad_node_id, "nomad node id was already set"); + } + + // Insert metrics + if let Some(install_complete_ts) = install_complete_ts { + insert_nomad_metrics( + input.cluster_id, + input.datacenter_id, + &input.provider_datacenter_id, + &input.datacenter_name_id, + nomad_join_ts, + install_complete_ts, + ); + } else { + tracing::warn!("missing install_complete_ts"); + } + + Ok(()) +} + +fn insert_nomad_metrics( + cluster_id: Uuid, + datacenter_id: Uuid, + provider_datacenter_id: &str, + datacenter_name_id: &str, + nomad_join_ts: i64, + install_complete_ts: i64, +) { + let dt = (nomad_join_ts - install_complete_ts) as f64 / 1000.0; + + metrics::NOMAD_JOIN_DURATION + .with_label_values(&[ + &cluster_id.to_string(), + &datacenter_id.to_string(), + provider_datacenter_id, + datacenter_name_id, + ]) + .observe(dt); +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct SetDrainCompleteInput { + server_id: Uuid, +} + +#[activity(SetDrainComplete)] +async fn set_drain_complete(ctx: &ActivityCtx, input: &SetDrainCompleteInput) -> GlobalResult<()> { + // Set as completed draining. Will be destroyed by `cluster-datacenter-scale` + sql_execute!( + [ctx] + " + UPDATE db_cluster.servers + SET drain_complete_ts = $2 + WHERE server_id = $1 + ", + input.server_id, + util::timestamp::now(), + ) + .await?; + + Ok(()) +} + +/// Finite state machine for handling server updates. +struct State { + draining: bool, + has_dns: bool, + is_tainted: bool, +} + +impl State { + /* ==== BINARY CONDITION DECOMPOSITION ==== + + // state + drain dns taint // available actions + 0 0 0 // drain, taint, dns create + 0 0 1 // drain + 0 1 0 // drain, taint, dns delete + 0 1 1 // drain, dns delete + 1 0 0 // undrain, taint, nomad drain complete + 1 0 1 // nomad drain complete + 1 1 0 // undrain, taint, dns delete, nomad drain complete + 1 1 1 // nomad drain complete + + destroy // always + drain // if !drain + undrain // if drain && !taint + taint // if !taint + dns create // if !dns && !drain && !taint + dns delete // if !(dns || drain && taint) + nomad registered // always + nomad drain complete // if drain + */ + async fn listen(&mut self, ctx: &mut WorkflowCtx) -> WorkflowResult

{ + // Determine which signals to listen to + let mut signals = vec![Destroy::NAME, NomadRegistered::NAME]; + + if self.draining { + signals.push(NomadDrainComplete::NAME); + + if !self.is_tainted { + signals.push(Undrain::NAME); + } + } else { + signals.push(Drain::NAME); + } + + if !self.is_tainted { + signals.push(Taint::NAME); + } + + if !self.has_dns && !self.draining && !self.is_tainted { + signals.push(DnsCreate::NAME); + } + + if !(self.has_dns || self.draining && self.is_tainted) { + signals.push(DnsDelete::NAME); + } + + let row = ctx.listen_any(&signals).await?; + let signal = Main::parse(&row.signal_name, row.body)?; + + // Update state + self.transition(&signal); + + Ok(signal) + } + + fn transition(&mut self, signal: &Main) { + match signal { + Main::Drain(_) => self.draining = true, + Main::Undrain(_) => self.draining = false, + Main::Taint(_) => self.is_tainted = true, + Main::DnsCreate(_) => self.has_dns = true, + Main::DnsDelete(_) => self.has_dns = false, + _ => {} + } + } +} + +impl Default for State { + fn default() -> Self { + State { + draining: false, + has_dns: true, + is_tainted: false, + } + } +} + +// Listen for linode provision signals +type ProvisionComplete = linode::workflows::server::ProvisionComplete; +type ProvisionFailed = linode::workflows::server::ProvisionFailed; +join_signal!(pub(crate) Linode, [ProvisionComplete, ProvisionFailed]); + +#[signal("cluster-server-drain")] +pub struct Drain {} + +#[signal("cluster-server-undrain")] +pub struct Undrain {} + +#[signal("cluster-server-taint")] +pub struct Taint {} + +#[signal("cluster-server-dns-create")] +pub struct DnsCreate {} + +#[signal("cluster-server-dns-delete")] +pub struct DnsDelete {} + +#[signal("cluster-server-destroy")] +pub struct Destroy {} + +#[signal("cluster-server-nomad-registered")] +pub struct NomadRegistered { + pub node_id: String, +} + +#[signal("cluster-server-nomad-drain-complete")] +pub struct NomadDrainComplete {} + +join_signal!( + Main, + [ + Drain, + Undrain, + Taint, + DnsCreate, + DnsDelete, + Destroy, + NomadRegistered, + NomadDrainComplete + ] +); diff --git a/svc/pkg/cluster/src/workflows/server/undrain.rs b/svc/pkg/cluster/src/workflows/server/undrain.rs new file mode 100644 index 000000000..80e5841ef --- /dev/null +++ b/svc/pkg/cluster/src/workflows/server/undrain.rs @@ -0,0 +1,101 @@ +use chirp_workflow::prelude::*; +use nomad_client::{ + apis::{configuration::Configuration, nodes_api}, + models, +}; +use rivet_operation::prelude::proto::backend::pkg::mm; +use serde_json::json; + +use crate::types::PoolType; + +lazy_static::lazy_static! { + static ref NOMAD_CONFIG: Configuration = nomad_util::new_config_from_env().unwrap(); +} + +#[derive(Debug, Serialize, Deserialize)] +pub(crate) struct Input { + pub datacenter_id: Uuid, + pub server_id: Uuid, + pub pool_type: PoolType, +} + +#[workflow] +pub(crate) async fn cluster_server_undrain( + ctx: &mut WorkflowCtx, + input: &Input, +) -> GlobalResult<()> { + match input.pool_type { + PoolType::Job => { + ctx.activity(UndrainNodeInput { + datacenter_id: input.datacenter_id, + server_id: input.server_id, + }) + .await?; + } + PoolType::Gg => { + ctx.tagged_signal( + &json!({ + "server_id": input.server_id, + }), + crate::workflows::server::DnsCreate {}, + ) + .await?; + } + PoolType::Ats => {} + } + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct UndrainNodeInput { + datacenter_id: Uuid, + server_id: Uuid, +} + +#[activity(UndrainNode)] +async fn undrain_node(ctx: &ActivityCtx, input: &UndrainNodeInput) -> GlobalResult<()> { + let (nomad_node_id,) = sql_fetch_one!( + [ctx, (Option,)] + " + SELECT nomad_node_id + FROM db_cluster.servers + WHERE server_id = $1 + ", + input.server_id, + ) + .await?; + + if let Some(nomad_node_id) = nomad_node_id { + nodes_api::update_node_drain( + &NOMAD_CONFIG, + &nomad_node_id, + models::NodeUpdateDrainRequest { + drain_spec: None, + mark_eligible: Some(true), + meta: None, + node_id: Some(nomad_node_id.clone()), + }, + None, + None, + None, + None, + None, + None, + None, + None, + None, + ) + .await?; + + // Allow new matchmaker requests to the node running on this server + msg!([ctx] mm::msg::nomad_node_closed_set(&nomad_node_id) { + datacenter_id: Some(input.datacenter_id.into()), + nomad_node_id: nomad_node_id.clone(), + is_closed: false, + }) + .await?; + } + + Ok(()) +} diff --git a/svc/pkg/cluster/standalone/datacenter-tls-renew/Cargo.toml b/svc/pkg/cluster/standalone/datacenter-tls-renew/Cargo.toml index 61b2cf47d..52feaf86a 100644 --- a/svc/pkg/cluster/standalone/datacenter-tls-renew/Cargo.toml +++ b/svc/pkg/cluster/standalone/datacenter-tls-renew/Cargo.toml @@ -7,16 +7,16 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } rivet-connection = { path = "../../../../../lib/connection" } rivet-health-checks = { path = "../../../../../lib/health-checks" } rivet-metrics = { path = "../../../../../lib/metrics" } -rivet-operation = { path = "../../../../../lib/operation/core" } rivet-runtime = { path = "../../../../../lib/runtime" } tokio = { version = "1.29", features = ["full"] } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } -cluster-datacenter-get = { path = "../../ops/datacenter-get" } +cluster = { path = "../.." } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" @@ -24,4 +24,3 @@ rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" default-features = false [dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/standalone/datacenter-tls-renew/src/lib.rs b/svc/pkg/cluster/standalone/datacenter-tls-renew/src/lib.rs index 6e4a489a5..80ea4d9d2 100644 --- a/svc/pkg/cluster/standalone/datacenter-tls-renew/src/lib.rs +++ b/svc/pkg/cluster/standalone/datacenter-tls-renew/src/lib.rs @@ -1,6 +1,7 @@ -use futures_util::FutureExt; -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; + +use cluster::types::TlsState; +use serde_json::json; // How much time before the cert expires to renew it const EXPIRE_PADDING: i64 = util::duration::days(30); @@ -10,67 +11,40 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { let client = chirp_client::SharedClient::from_env(pools.clone())? .wrap_new("cluster-datacenter-tls-renew"); let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "cluster-datacenter-tls-renew".into(), - std::time::Duration::from_secs(60), + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); - - let updated_datacenter_ids = rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - let ctx = ctx.base(); - - async move { - // Check for expired rows - let datacenters = sql_fetch_all!( - [ctx, (Uuid,), @tx tx] - " - SELECT - datacenter_id - FROM db_cluster.datacenter_tls - WHERE - state = $1 AND - expire_ts < $2 - FOR UPDATE - ", - backend::cluster::TlsState::Active as i64, - util::timestamp::now() + EXPIRE_PADDING, - ) - .await? - .into_iter() - .map(|(datacenter_id,)| datacenter_id) - .collect::>(); - - // Set as renewing - for datacenter_id in &datacenters { - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.datacenter_tls - SET state = $2 - WHERE datacenter_id = $1 - ", - datacenter_id, - backend::cluster::TlsState::Renewing as i64, - ) - .await?; - } - - Ok(datacenters) - } - .boxed() - }) + "cluster-datacenter-tls-renew", + ) .await?; + let updated_datacenter_ids = sql_fetch_all!( + [ctx, (Uuid,)] + " + UPDATE db_cluster.datacenter_tls + SET state2 = $3 + FROM db_cluster.datacenter_tls + WHERE + state2 = $1 AND + expire_ts < $2 + RETURNING datacenter_id + ", + serde_json::to_string(&TlsState::Active)?, + util::timestamp::now() + EXPIRE_PADDING, + serde_json::to_string(&TlsState::Renewing)?, + ) + .await? + .into_iter() + .map(|(datacenter_id,)| datacenter_id) + .collect::>(); + for datacenter_id in updated_datacenter_ids { - msg!([ctx] cluster::msg::datacenter_tls_issue(datacenter_id) { - datacenter_id: Some(datacenter_id.into()), - renew: true, - }) + ctx.tagged_signal( + &json!({ + "datacenter_id": datacenter_id, + }), + cluster::workflows::datacenter::TlsRenew {}, + ) .await?; } diff --git a/svc/pkg/cluster/standalone/datacenter-tls-renew/src/main.rs b/svc/pkg/cluster/standalone/datacenter-tls-renew/src/main.rs index 9ae0d694d..e554bbdbf 100644 --- a/svc/pkg/cluster/standalone/datacenter-tls-renew/src/main.rs +++ b/svc/pkg/cluster/standalone/datacenter-tls-renew/src/main.rs @@ -1,6 +1,6 @@ use std::time::Duration; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; fn main() -> GlobalResult<()> { rivet_runtime::run(start()).unwrap() diff --git a/svc/pkg/cluster/standalone/datacenter-tls-renew/tests/integration.rs b/svc/pkg/cluster/standalone/datacenter-tls-renew/tests/integration.rs index 2886bdbae..f21af06e5 100644 --- a/svc/pkg/cluster/standalone/datacenter-tls-renew/tests/integration.rs +++ b/svc/pkg/cluster/standalone/datacenter-tls-renew/tests/integration.rs @@ -1,4 +1,4 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; #[tokio::test(flavor = "multi_thread")] async fn basic() { @@ -12,8 +12,10 @@ async fn basic() { .with_span_events(tracing_subscriber::fmt::format::FmtSpan::NONE) .init(); - let _ctx = TestCtx::from_env("cluster-gc-test").await.unwrap(); - let _pools = rivet_pools::from_env("cluster-gc-test").await.unwrap(); + let _ctx = TestCtx::from_env("cluster-datacenter-tls-renew-test").await; + let _pools = rivet_pools::from_env("cluster-datacenter-tls-renew-test") + .await + .unwrap(); // TODO: } diff --git a/svc/pkg/cluster/standalone/default-update/Cargo.toml b/svc/pkg/cluster/standalone/default-update/Cargo.toml index 4fb14ac06..8aa8c1cd2 100644 --- a/svc/pkg/cluster/standalone/default-update/Cargo.toml +++ b/svc/pkg/cluster/standalone/default-update/Cargo.toml @@ -7,10 +7,9 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -prost = "0.10" -rivet-connection = { path = "../../../../../lib/connection" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } reqwest = "0.11" +rivet-connection = { path = "../../../../../lib/connection" } rivet-pools = { path = "../../../../../lib/pools" } serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" @@ -18,11 +17,8 @@ tokio = { version = "1.29", features = ["full"] } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } uuid = { version = "1", features = ["v4"] } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } -cluster-get = { path = "../../ops/get" } -cluster-datacenter-get = { path = "../../ops/datacenter-get" } -cluster-datacenter-list = { path = "../../ops/datacenter-list" } +cluster = { path = "../.." } [dev-dependencies] chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/standalone/default-update/src/lib.rs b/svc/pkg/cluster/standalone/default-update/src/lib.rs index 4a9a1fe78..44b3977b9 100644 --- a/svc/pkg/cluster/standalone/default-update/src/lib.rs +++ b/svc/pkg/cluster/standalone/default-update/src/lib.rs @@ -1,9 +1,8 @@ use std::collections::HashMap; -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; use serde::Deserialize; -use uuid::Uuid; +use serde_json::json; #[derive(Deserialize)] struct Cluster { @@ -28,10 +27,10 @@ enum Provider { Linode, } -impl From for backend::cluster::Provider { - fn from(value: Provider) -> backend::cluster::Provider { +impl From for cluster::types::Provider { + fn from(value: Provider) -> cluster::types::Provider { match value { - Provider::Linode => backend::cluster::Provider::Linode, + Provider::Linode => cluster::types::Provider::Linode, } } } @@ -55,12 +54,12 @@ enum PoolType { Ats, } -impl From for backend::cluster::PoolType { - fn from(value: PoolType) -> backend::cluster::PoolType { +impl From for cluster::types::PoolType { + fn from(value: PoolType) -> cluster::types::PoolType { match value { - PoolType::Job => backend::cluster::PoolType::Job, - PoolType::Gg => backend::cluster::PoolType::Gg, - PoolType::Ats => backend::cluster::PoolType::Ats, + PoolType::Job => cluster::types::PoolType::Job, + PoolType::Gg => cluster::types::PoolType::Gg, + PoolType::Ats => cluster::types::PoolType::Ats, } } } @@ -70,9 +69,9 @@ struct Hardware { name: String, } -impl From for backend::cluster::Hardware { - fn from(value: Hardware) -> backend::cluster::Hardware { - backend::cluster::Hardware { +impl From for cluster::types::Hardware { + fn from(value: Hardware) -> cluster::types::Hardware { + cluster::types::Hardware { provider_hardware: value.name, } } @@ -86,13 +85,13 @@ enum BuildDeliveryMethod { S3Direct, } -impl From for backend::cluster::BuildDeliveryMethod { - fn from(value: BuildDeliveryMethod) -> backend::cluster::BuildDeliveryMethod { +impl From for cluster::types::BuildDeliveryMethod { + fn from(value: BuildDeliveryMethod) -> cluster::types::BuildDeliveryMethod { match value { BuildDeliveryMethod::TrafficServer => { - backend::cluster::BuildDeliveryMethod::TrafficServer + cluster::types::BuildDeliveryMethod::TrafficServer } - BuildDeliveryMethod::S3Direct => backend::cluster::BuildDeliveryMethod::S3Direct, + BuildDeliveryMethod::S3Direct => cluster::types::BuildDeliveryMethod::S3Direct, } } } @@ -103,16 +102,12 @@ pub async fn run_from_env(use_autoscaler: bool) -> GlobalResult<()> { let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("cluster-default-update"); let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "cluster-default-update".into(), - std::time::Duration::from_secs(60), + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); + "cluster-default-update", + ) + .await?; // Read config from env let Some(config_json) = util::env::var("RIVET_DEFAULT_CLUSTER_CONFIG").ok() else { @@ -121,64 +116,56 @@ pub async fn run_from_env(use_autoscaler: bool) -> GlobalResult<()> { }; let config = serde_json::from_str::(&config_json)?; - let taint = util::env::var("RIVET_TAINT_DEFAULT_CLUSTER") - .ok() - .unwrap_or_else(|| "0".to_string()) - == "1"; - // HACK: When deploying both monolith worker and this service for the first time, there is a race // condition which might result in the message being published from here but not caught by // monolith-worker, resulting in nothing happening. tokio::time::sleep(std::time::Duration::from_secs(5)).await; - let cluster_id = util_cluster::default_cluster_id(); + let cluster_id = cluster::util::default_cluster_id(); let (cluster_res, datacenter_list_res) = tokio::try_join!( // Check if cluster already exists - op!([ctx] cluster_get { - cluster_ids: vec![cluster_id.into()], + ctx.op(cluster::ops::get::Input { + cluster_ids: vec![cluster_id], }), - op!([ctx] cluster_datacenter_list { - cluster_ids: vec![cluster_id.into()], + ctx.op(cluster::ops::datacenter::list::Input { + cluster_ids: vec![cluster_id], }), )?; // Get all datacenters let cluster = unwrap!(datacenter_list_res.clusters.first()); - let datacenters_res = op!([ctx] cluster_datacenter_get { + let datacenters_res = ctx.op(cluster::ops::datacenter::get::Input { datacenter_ids: cluster.datacenter_ids.clone(), - }) - .await?; + }).await?; if cluster_res.clusters.is_empty() { tracing::warn!("creating default cluster"); - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), + ctx.tagged_workflow(&json!({ + "cluster_id": cluster_id, + }), cluster::workflows::cluster::Input { + cluster_id, name_id: config.name_id.clone(), owner_team_id: None, - }) - .await?; + }).await?; } for existing_datacenter in &datacenters_res.datacenters { - let datacenter_id = unwrap_ref!(existing_datacenter.datacenter_id).as_uuid(); - if !config .datacenters .iter() - .any(|(_, dc)| dc.datacenter_id == datacenter_id) + .any(|(_, dc)| dc.datacenter_id == existing_datacenter.datacenter_id) { // TODO: Delete datacenters } } for (name_id, datacenter) in config.datacenters { - let datacenter_id_proto = datacenter.datacenter_id.into(); let existing_datacenter = datacenters_res .datacenters .iter() - .any(|dc| dc.datacenter_id == Some(datacenter_id_proto)); + .any(|dc| dc.datacenter_id == datacenter.datacenter_id); // Update existing datacenter if existing_datacenter { @@ -197,8 +184,8 @@ pub async fn run_from_env(use_autoscaler: bool) -> GlobalResult<()> { } }; - cluster::msg::datacenter_update::PoolUpdate { - pool_type: Into::::into(pool_type) as i32, + cluster::types::PoolUpdate { + pool_type: pool_type.into(), hardware: pool .hardware .into_iter() @@ -212,28 +199,29 @@ pub async fn run_from_env(use_autoscaler: bool) -> GlobalResult<()> { }) .collect::>(); - msg!([ctx] @wait cluster::msg::datacenter_update(datacenter.datacenter_id) { - datacenter_id: Some(datacenter_id_proto), + ctx.tagged_signal(&json!({ + "datacenter_id": datacenter.datacenter_id, + }), cluster::workflows::datacenter::Update { pools: new_pools, prebakes_enabled: Some(datacenter.prebakes_enabled), - }) - .await?; + }).await?; } // Create new datacenter else { - msg!([ctx] @wait cluster::msg::datacenter_create(datacenter.datacenter_id) { - datacenter_id: Some(datacenter_id_proto), - cluster_id: Some(cluster_id.into()), + ctx.tagged_signal(&json!({ + "cluster_id": cluster_id, + }), cluster::workflows::cluster::DatacenterCreate { + datacenter_id: datacenter.datacenter_id, name_id, display_name: datacenter.display_name, - - provider: Into::::into(datacenter.provider) as i32, + + provider: datacenter.provider.into(), provider_datacenter_id: datacenter.provider_datacenter_name, provider_api_token: None, - + pools: datacenter.pools.into_iter().map(|(pool_type, pool)| { - backend::cluster::Pool { - pool_type: Into::::into(pool_type) as i32, + cluster::types::Pool { + pool_type: pool_type.into(), hardware: pool.hardware.into_iter().map(Into::into).collect::>(), desired_count: pool.desired_count, min_count: pool.min_count, @@ -241,27 +229,10 @@ pub async fn run_from_env(use_autoscaler: bool) -> GlobalResult<()> { drain_timeout: pool.drain_timeout, } }).collect::>(), - - build_delivery_method: Into::::into(datacenter.build_delivery_method) as i32, + + build_delivery_method: datacenter.build_delivery_method.into(), prebakes_enabled: datacenter.prebakes_enabled, - }) - .await?; - } - - // TODO: Both this message and datacenter-create/datacenter-update (above) publish datacenter-scale. - // This results in double provisioning until datacenter-scale is published again, cleaning up the - // excess. - // Taint datacenter - if taint { - let request_id = Uuid::new_v4(); - msg!([ctx] @wait cluster::msg::server_taint(request_id) { - filter: Some(backend::cluster::ServerFilter { - filter_datacenter_ids: true, - datacenter_ids: vec![datacenter_id_proto], - ..Default::default() - }), - }) - .await?; + }).await?; } } diff --git a/svc/pkg/cluster/standalone/default-update/src/main.rs b/svc/pkg/cluster/standalone/default-update/src/main.rs index 8fad08930..8e4997a64 100644 --- a/svc/pkg/cluster/standalone/default-update/src/main.rs +++ b/svc/pkg/cluster/standalone/default-update/src/main.rs @@ -1,4 +1,4 @@ -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; #[tokio::main] async fn main() -> GlobalResult<()> { @@ -10,7 +10,7 @@ async fn main() -> GlobalResult<()> { // TODO: When running bolt up, this service gets created first before `cluster-worker` so the messages // sent from here are received but effectively forgotten because `cluster-worker` gets restarted - // immediately afterwards. + // immediately afterwards. This server will be replaced with a bolt infra step tokio::time::sleep(std::time::Duration::from_secs(3)).await; cluster_default_update::run_from_env(false).await diff --git a/svc/pkg/cluster/standalone/fix-tls/Cargo.toml b/svc/pkg/cluster/standalone/fix-tls/Cargo.toml deleted file mode 100644 index 7bea53a34..000000000 --- a/svc/pkg/cluster/standalone/fix-tls/Cargo.toml +++ /dev/null @@ -1,47 +0,0 @@ -[package] -name = "cluster-fix-tls" -version = "0.0.1" -edition = "2021" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -rivet-connection = { path = "../../../../../lib/connection" } -rivet-runtime = { path = "../../../../../lib/runtime" } -tokio = { version = "1.29", features = ["full"] } -tracing = "0.1" -tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } - -acme-lib = "0.9" -anyhow = "1.0" -chrono = "0.4" -cloudflare = "0.10.1" -http = "0.2" -include_dir = "0.7.3" -indoc = "1.0" -lazy_static = "1.4" -maplit = "1.0" -nomad-util = { path = "../../../../../lib/nomad-util" } -openssl = "0.10.63" -rivet-convert = { path = "../../../../../lib/convert" } -rivet-health-checks = { path = "../../../../../lib/health-checks" } -rivet-metrics = { path = "../../../../../lib/metrics" } -s3-util = { path = "../../../../../lib/s3-util" } -serde_yaml = "0.9" -ssh2 = "0.9.4" -thiserror = "1.0" -trust-dns-resolver = { version = "0.23.2", features = ["dns-over-native-tls"] } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } - -cluster-datacenter-get = { path = "../../ops/datacenter-get" } -cluster-datacenter-list = { path = "../../ops/datacenter-list" } -cluster-datacenter-topology-get = { path = "../../ops/datacenter-topology-get" } -linode-instance-type-get = { path = "../../../linode/ops/instance-type-get" } -linode-server-destroy = { path = "../../../linode/ops/server-destroy" } -linode-server-provision = { path = "../../../linode/ops/server-provision" } -token-create = { path = "../../../token/ops/create" } - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/standalone/fix-tls/Service.toml b/svc/pkg/cluster/standalone/fix-tls/Service.toml deleted file mode 100644 index 4ab0a42ff..000000000 --- a/svc/pkg/cluster/standalone/fix-tls/Service.toml +++ /dev/null @@ -1,13 +0,0 @@ -[service] -name = "cluster-fix-tls" - -[runtime] -kind = "rust" - -[secrets] -"rivet/api_traefik_provider/token" = {} -"cloudflare/terraform/auth_token" = { optional = true } -"ssh/server/private_key_openssh" = {} - -[headless] -singleton = true diff --git a/svc/pkg/cluster/standalone/fix-tls/src/lib.rs b/svc/pkg/cluster/standalone/fix-tls/src/lib.rs deleted file mode 100644 index cbfa37ccc..000000000 --- a/svc/pkg/cluster/standalone/fix-tls/src/lib.rs +++ /dev/null @@ -1,335 +0,0 @@ -use acme_lib::{ - create_p384_key, - persist::{MemoryPersist, Persist, PersistKey, PersistKind}, - Account, Certificate, Directory, DirectoryUrl, -}; -use cloudflare::{endpoints as cf, framework as cf_framework, framework::async_api::ApiClient}; -use futures_util::StreamExt; -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; -use tokio::task; -use trust_dns_resolver::{ - config::{ResolverConfig, ResolverOpts}, - error::ResolveErrorKind, - TokioAsyncResolver, -}; - -#[derive(thiserror::Error, Debug)] -#[error("cloudflare: {source}")] -pub struct CloudflareError { - #[from] - source: anyhow::Error, -} - -const ENCRYPT_EMAIL: &str = "letsencrypt@rivet.gg"; - -#[tracing::instrument(skip_all)] -pub async fn run_from_env(ts: i64) -> GlobalResult<()> { - tracing::warn!("disabled for now"); - return Ok(()); - - let pools = rivet_pools::from_env("cluster-fix-tls").await?; - let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("cluster-fix-tls"); - let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "cluster-fix-tls".into(), - std::time::Duration::from_secs(60), - rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); - - let datacenter_ids = vec!["5767a802-5c7c-4563-a266-33c014f7e244"] - .into_iter() - .map(|x| Uuid::parse_str(x).unwrap()); - - for id in datacenter_ids { - let ctx = ctx.clone(); - tokio::spawn(async move { - match run_for_datacenter(ctx, id).await { - Ok(_) => { - tracing::info!(?id, "datacenter done 2"); - } - Err(err) => { - tracing::error!(?id, ?err, "datacenter failed 2"); - } - } - }); - } - - std::future::pending::<()>().await; - - Ok(()) -} - -async fn run_for_datacenter(ctx: OperationContext<()>, datacenter_id: Uuid) -> GlobalResult<()> { - let renew = false; - - // Create CF client - let cf_token = util::env::read_secret(&["cloudflare", "terraform", "auth_token"]).await?; - let client = cf_framework::async_api::Client::new( - cf_framework::auth::Credentials::UserAuthToken { token: cf_token }, - Default::default(), - cf_framework::Environment::Production, - ) - .map_err(CloudflareError::from)?; - - // Fetch ACME account registration - let account = acme_account().await?; - - let base_zone_id = unwrap!( - util::env::cloudflare::zone::main::id(), - "dns not configured" - ); - let job_zone_id = unwrap!(util::env::cloudflare::zone::job::id(), "dns not configured"); - let domain_main = unwrap!(util::env::domain_main(), "dns not enabled"); - let domain_job = unwrap!(util::env::domain_job(), "dns not enabled"); - - // NOTE: We don't use try_join because these run in parallel, the dns record needs to be deleted for each - // order upon failure - let job_cert = order( - &client, - renew, - job_zone_id, - &account, - domain_job, - vec![ - // TODO: Remove this - format!("i-see-you-skid.{domain_job}"), - format!("*.lobby.{datacenter_id}.{domain_job}"), - format!("*.{datacenter_id}.{domain_job}"), - ], - ) - .await?; - - sql_execute!( - [ctx] - " - UPDATE db_cluster.datacenter_tls - SET - gg_cert_pem = $2, - gg_private_key_pem = $3, - job_cert_pem = $4, - job_private_key_pem = $5, - state = $6, - expire_ts = $7 - WHERE datacenter_id = $1 - ", - datacenter_id, - "N/A", - "N/A", - job_cert.certificate(), - job_cert.private_key(), - backend::cluster::TlsState::Active as i64, - util::timestamp::now() + util::duration::days(job_cert.valid_days_left()), - ) - .await?; - - tracing::info!("done"); - - Ok(()) -} - -async fn acme_account() -> GlobalResult> { - let url = match util::env::var("TLS_ACME_DIRECTORY")?.as_str() { - "lets_encrypt" => DirectoryUrl::LetsEncrypt, - "lets_encrypt_staging" => DirectoryUrl::LetsEncryptStaging, - x => bail!(format!("unknown ACME directory: {x}")), - }; - - let persist = MemoryPersist::new(); - - // Write account private key (from terraform) to persistence - let pem_key = PersistKey::new( - ENCRYPT_EMAIL, - PersistKind::AccountPrivateKey, - "acme_account", - ); - let pem = util::env::var("TLS_ACME_ACCOUNT_PRIVATE_KEY_PEM")?; - persist.put(&pem_key, pem.as_bytes())?; - - // Get ACME account info - let acc = tokio::task::spawn_blocking(move || { - // Initialize ACME directory - let dir = Directory::from_url(persist, url)?; - - tracing::info!("fetching account"); - dir.account(ENCRYPT_EMAIL) - }) - .await??; - - Ok(acc) -} - -// TODO: This function contains both blocking calls that cannot be shared between threads and async calls. -// Maybe theres a way to defer the blocking calls somehow -async fn order( - client: &cf_framework::async_api::Client, - renew: bool, - zone_id: &str, - account: &Account

, - common_name: &str, - subject_alternative_names: Vec, -) -> GlobalResult { - tracing::info!(cn=%common_name, "creating order"); - - let mut order = task::block_in_place(|| { - account.new_order( - common_name, - &subject_alternative_names - .iter() - .map(|s| s.as_str()) - .collect::>(), - ) - })?; - - // When not renewing, if the ownership of the domain(s) have already been authorized in a previous order - // we might be able to skip validation. The ACME API provider decides. - let order_csr = if let Some(order_csr) = renew.then(|| order.confirm_validations()).flatten() { - order_csr - } else { - loop { - tracing::info!(%common_name, "fetching authorizations"); - let auths = task::block_in_place(|| order.authorizations())?; - - // Run authorizations in parallel - let results = futures_util::stream::iter(auths.into_iter().map(|auth| { - async move { - let challenge = auth.dns_challenge(); - let proof = challenge.dns_proof(); - - let hostname = format!("_acme-challenge.{}", auth.api_auth().identifier.value); - let dns_record_id = - create_dns_record(client, zone_id, &hostname, &proof).await?; - - let try_block = async { - // Wait for DNS to propagate - poll_txt_dns(&hostname, &proof).await?; - - tracing::info!(%hostname, "validating authorization"); - task::block_in_place(|| challenge.validate(5000))?; - - GlobalResult::Ok(()) - } - .await; - - // Delete regardless of success of the above try block - // match delete_dns_record(client, zone_id, &dns_record_id).await { - // Ok(_) => { - // - // } - // Err(err) => { - // tracing::error!(?zone_id, ?dns_record_id, ?hostname, ?err, "failed to delete dns record"); - // } - // } - - try_block - } - })) - .buffer_unordered(4) - .collect::>() - .await; - - // Handle errors only after all futures have completed so that we ensure all dns records are deleted - for res in results { - res?; - } - - tracing::info!("refreshing order"); - task::block_in_place(|| order.refresh())?; - - if let Some(order_csr) = order.confirm_validations() { - break order_csr; - } - } - }; - - tracing::info!("order validated"); - - // Submit the CSR - let cert_pri = create_p384_key(); - let ord_cert = task::block_in_place(|| order_csr.finalize_pkey(cert_pri, 5000))?; - let cert = task::block_in_place(|| ord_cert.download_and_save_cert())?; - - tracing::info!("order finalized"); - - Ok(cert) -} - -async fn create_dns_record( - client: &cf_framework::async_api::Client, - zone_id: &str, - record_name: &str, - content: &str, -) -> GlobalResult { - tracing::info!(%record_name, "creating dns record"); - - let create_record_res = client - .request(&cf::dns::CreateDnsRecord { - zone_identifier: zone_id, - params: cf::dns::CreateDnsRecordParams { - name: record_name, - content: cf::dns::DnsContent::TXT { - content: content.to_string(), - }, - proxied: Some(false), - ttl: Some(60), - priority: None, - }, - }) - .await?; - - Ok(create_record_res.result.id) -} - -async fn delete_dns_record( - client: &cf_framework::async_api::Client, - zone_id: &str, - record_id: &str, -) -> GlobalResult<()> { - tracing::info!(%record_id, "deleting dns record"); - - client - .request(&cf::dns::DeleteDnsRecord { - zone_identifier: zone_id, - identifier: record_id, - }) - .await?; - - Ok(()) -} - -async fn poll_txt_dns(hostname: &str, content: &str) -> GlobalResult<()> { - // Because the dns resolver has its own internal cache, we create a new one for each poll function call - // so that clearing cache does not affect other concurrent txt lookup calls - let dns_resolver = - TokioAsyncResolver::tokio(ResolverConfig::cloudflare_tls(), ResolverOpts::default()); - - // Fully qualified domain name lookups are faster - let fqdn = format!("{hostname}."); - - // Retry DNS until the TXT record shows up - for attempt in 1..=100 { - tokio::time::sleep(std::time::Duration::from_secs(2)).await; - - tracing::info!(%attempt, %fqdn, "attempting to resolve dns"); - - dns_resolver.clear_cache(); - - match dns_resolver.txt_lookup(&fqdn).await { - Ok(res) => { - if res.iter().any(|record| record.to_string() == content) { - return Ok(()); - } - } - // Retry - Err(err) if matches!(err.kind(), ResolveErrorKind::NoRecordsFound { .. }) => {} - Err(err) => return Err(err.into()), - } - } - - bail!("dns not resolved"); -} diff --git a/svc/pkg/cluster/standalone/fix-tls/src/main.rs b/svc/pkg/cluster/standalone/fix-tls/src/main.rs deleted file mode 100644 index 4dfb986d6..000000000 --- a/svc/pkg/cluster/standalone/fix-tls/src/main.rs +++ /dev/null @@ -1,11 +0,0 @@ -use rivet_operation::prelude::*; - -fn main() -> GlobalResult<()> { - rivet_runtime::run(start()).unwrap() -} - -async fn start() -> GlobalResult<()> { - cluster_fix_tls::run_from_env(util::timestamp::now()).await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/standalone/fix-tls/tests/integration.rs b/svc/pkg/cluster/standalone/fix-tls/tests/integration.rs deleted file mode 100644 index fd3a5a30c..000000000 --- a/svc/pkg/cluster/standalone/fix-tls/tests/integration.rs +++ /dev/null @@ -1,15 +0,0 @@ -use chirp_worker::prelude::*; - -use ::cluster_fix_tls::run_from_env; - -#[tokio::test(flavor = "multi_thread")] -async fn basic() { - tracing_subscriber::fmt() - .json() - .with_max_level(tracing::Level::INFO) - .with_span_events(tracing_subscriber::fmt::format::FmtSpan::NONE) - .init(); - - // TODO: - run_from_env(util::timestamp::now()).await.unwrap(); -} diff --git a/svc/pkg/cluster/standalone/gc/Cargo.toml b/svc/pkg/cluster/standalone/gc/Cargo.toml index 242b1b67f..c754a4bb3 100644 --- a/svc/pkg/cluster/standalone/gc/Cargo.toml +++ b/svc/pkg/cluster/standalone/gc/Cargo.toml @@ -7,16 +7,16 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } rivet-connection = { path = "../../../../../lib/connection" } rivet-health-checks = { path = "../../../../../lib/health-checks" } rivet-metrics = { path = "../../../../../lib/metrics" } -rivet-operation = { path = "../../../../../lib/operation/core" } rivet-runtime = { path = "../../../../../lib/runtime" } tokio = { version = "1.29", features = ["full"] } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } -cluster-datacenter-get = { path = "../../ops/datacenter-get" } +cluster = { path = "../.." } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" @@ -24,5 +24,3 @@ rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" default-features = false [dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } diff --git a/svc/pkg/cluster/standalone/gc/src/lib.rs b/svc/pkg/cluster/standalone/gc/src/lib.rs index b520f539e..bb64e769d 100644 --- a/svc/pkg/cluster/standalone/gc/src/lib.rs +++ b/svc/pkg/cluster/standalone/gc/src/lib.rs @@ -1,11 +1,15 @@ +use std::convert::TryInto; + +use chirp_workflow::prelude::*; +use cluster::types::PoolType; use futures_util::FutureExt; -use proto::backend::{self, pkg::*}; -use rivet_operation::prelude::*; +use serde_json::json; #[derive(sqlx::FromRow)] struct ServerRow { server_id: Uuid, datacenter_id: Uuid, + pool_type2: Option>, pool_type: i64, drain_ts: i64, } @@ -14,33 +18,34 @@ struct ServerRow { pub async fn run_from_env(ts: i64, pools: rivet_pools::Pools) -> GlobalResult<()> { let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("cluster-gc"); let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "cluster-gc".into(), - std::time::Duration::from_secs(60), + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); + "cluster-gc", + ) + .await?; let datacenter_ids = rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - let ctx = ctx.base(); + let ctx = ctx.clone(); async move { + let pool_types = [ + serde_json::to_string(&PoolType::Gg)?, + serde_json::to_string(&PoolType::Ats)?, + ]; + // Select all draining gg and ats servers let servers = sql_fetch_all!( [ctx, ServerRow, @tx tx] " - SELECT server_id, datacenter_id, pool_type, drain_ts + SELECT server_id, datacenter_id, pool_type, pool_type2, drain_ts FROM db_cluster.servers WHERE - pool_type = ANY($1) AND + pool_type2 = ANY($1) AND cloud_destroy_ts IS NULL AND drain_ts IS NOT NULL ", - &[backend::cluster::PoolType::Gg as i64, backend::cluster::PoolType::Ats as i64], + &pool_types, ts, ) .await?; @@ -49,26 +54,31 @@ pub async fn run_from_env(ts: i64, pools: rivet_pools::Pools) -> GlobalResult<() return Ok(Vec::new()); } - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: servers - .iter() - .map(|server| server.datacenter_id.into()) - .collect::>(), - }) - .await?; + let datacenters_res = ctx + .op(cluster::ops::datacenter::get::Input { + datacenter_ids: servers + .iter() + .map(|server| server.datacenter_id) + .collect::>(), + }) + .await?; let drained_servers = servers .into_iter() .map(|server| { - let dc_id_proto = Some(server.datacenter_id.into()); + let pool_type = if let Some(pool_type) = server.pool_type2.clone() { + pool_type.0 + } else { + server.pool_type.try_into()? + }; let datacenter = unwrap!(datacenters_res .datacenters .iter() - .find(|dc| dc.datacenter_id == dc_id_proto)); + .find(|dc| dc.datacenter_id == server.datacenter_id)); let pool = unwrap!(datacenter .pools .iter() - .find(|pool| pool.pool_type == server.pool_type as i32)); + .find(|pool| pool.pool_type == pool_type)); let drain_completed = server.drain_ts < ts - pool.drain_timeout as i64; tracing::info!( @@ -118,9 +128,12 @@ pub async fn run_from_env(ts: i64, pools: rivet_pools::Pools) -> GlobalResult<() // Scale for datacenter_id in datacenter_ids { - msg!([ctx] cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: Some(datacenter_id.into()), - }) + ctx.tagged_signal( + &json!({ + "datacenter_id": datacenter_id, + }), + cluster::workflows::datacenter::Scale {}, + ) .await?; } diff --git a/svc/pkg/cluster/standalone/gc/src/main.rs b/svc/pkg/cluster/standalone/gc/src/main.rs index 142bfd510..6657f9830 100644 --- a/svc/pkg/cluster/standalone/gc/src/main.rs +++ b/svc/pkg/cluster/standalone/gc/src/main.rs @@ -1,6 +1,6 @@ use std::time::Duration; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; fn main() -> GlobalResult<()> { rivet_runtime::run(start()).unwrap() diff --git a/svc/pkg/cluster/standalone/gc/tests/integration.rs b/svc/pkg/cluster/standalone/gc/tests/integration.rs index 19fb4187c..53c045e04 100644 --- a/svc/pkg/cluster/standalone/gc/tests/integration.rs +++ b/svc/pkg/cluster/standalone/gc/tests/integration.rs @@ -1,6 +1,8 @@ use ::cluster_gc::run_from_env; -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; +use serde_json::json; + +use cluster::types::{BuildDeliveryMethod, Hardware, Pool, PoolType, Provider}; const DRAIN_TIMEOUT: i64 = 1000 * 60 * 60; @@ -16,24 +18,10 @@ async fn basic() { .with_span_events(tracing_subscriber::fmt::format::FmtSpan::NONE) .init(); - let ctx = TestCtx::from_env("cluster-gc-test").await.unwrap(); + let ctx = TestCtx::from_env("cluster-gc-test").await; let pools = rivet_pools::from_env("cluster-gc-test").await.unwrap(); - let server_id = Uuid::new_v4(); - let datacenter_id = Uuid::new_v4(); - let cluster_id = Uuid::new_v4(); - - let (dc_pools, provider) = setup(&ctx, server_id, datacenter_id, cluster_id).await; - - msg!([ctx] cluster::msg::server_provision(server_id) { - datacenter_id: Some(datacenter_id.into()), - server_id: Some(server_id.into()), - pool_type: dc_pools.first().unwrap().pool_type, - provider: provider as i32, - tags: vec!["test".to_string()], - }) - .await - .unwrap(); + let (server_id, _) = setup(&ctx).await; // Wait for server to have an ip loop { @@ -73,77 +61,117 @@ async fn basic() { ) .await .unwrap(); - msg!([ctx] @wait cluster::msg::server_drain(server_id) { - server_id: Some(server_id.into()), - }) + + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + cluster::workflows::server::Drain {}, + ) .await .unwrap(); - let mut sub = subscribe!([ctx] cluster::msg::server_destroy(server_id)) - .await - .unwrap(); - // Run GC let ts = util::timestamp::now() + DRAIN_TIMEOUT + 1; run_from_env(ts, pools).await.unwrap(); - // Check that destroy message was sent - sub.next().await.unwrap(); + // Wait for server to be completely drained + loop { + tokio::time::sleep(std::time::Duration::from_secs(5)).await; + + let (exists,) = sql_fetch_one!( + [ctx, (bool,)] + " + SELECT EXISTS ( + SELECT 1 + FROM db_cluster.servers + WHERE + server_id = $1 AND + drain_complete_ts IS NOT NULL + ) + ", + server_id, + ) + .await + .unwrap(); + + if exists { + break; + } + } // Clean up afterwards so we don't litter - msg!([ctx] @wait cluster::msg::server_destroy(server_id) { - server_id: Some(server_id.into()), - force: false, - }) + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + cluster::workflows::server::Destroy {}, + ) .await .unwrap(); } -async fn setup( - ctx: &TestCtx, - server_id: Uuid, - datacenter_id: Uuid, - cluster_id: Uuid, -) -> (Vec, backend::cluster::Provider) { - let pool_type = backend::cluster::PoolType::Gg as i32; - let pools = vec![backend::cluster::Pool { - pool_type, - hardware: vec![backend::cluster::Hardware { - provider_hardware: util_cluster::test::LINODE_HARDWARE.to_string(), +async fn setup(ctx: &TestCtx) -> (Uuid, Uuid) { + let server_id = Uuid::new_v4(); + let datacenter_id = Uuid::new_v4(); + let cluster_id = Uuid::new_v4(); + + let pool_type = PoolType::Gg; + let pools = vec![Pool { + pool_type: pool_type.clone(), + hardware: vec![Hardware { + provider_hardware: cluster::util::test::LINODE_HARDWARE.to_string(), }], desired_count: 0, min_count: 0, max_count: 0, drain_timeout: DRAIN_TIMEOUT as u64, }]; - let provider = backend::cluster::Provider::Linode; - - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) + let provider = Provider::Linode; + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) .await .unwrap(); - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - display_name: util::faker::ident(), + let mut create_sub = ctx + .subscribe::(&json!({ + "datacenter_id": datacenter_id, + })) + .await + .unwrap(); + ctx.tagged_signal( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id, + name_id: util::faker::ident(), + display_name: util::faker::ident(), - provider: provider as i32, - provider_datacenter_id: "us-southeast".to_string(), - provider_api_token: None, + provider: provider.clone(), + provider_datacenter_id: "us-southeast".to_string(), + provider_api_token: None, - pools: pools.clone(), + pools: pools.clone(), - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) + build_delivery_method: BuildDeliveryMethod::TrafficServer, + prebakes_enabled: false, + }, + ) .await .unwrap(); + create_sub.next().await.unwrap(); + // Write new server to db sql_execute!( [ctx] @@ -151,18 +179,34 @@ async fn setup( INSERT INTO db_cluster.servers ( server_id, datacenter_id, - pool_type, - create_ts + pool_type2, + create_ts, + -- Backwards compatibility + pool_type ) - VALUES ($1, $2, $3, $4) + VALUES ($1, $2, $3, $4, 0) ", server_id, datacenter_id, - pool_type as i64, + serde_json::to_string(&pool_type).unwrap(), util::timestamp::now(), ) .await .unwrap(); - (pools, provider) + ctx.tagged_signal( + &json!({ + "datacenter_id": datacenter_id, + }), + cluster::workflows::datacenter::ServerCreate { + server_id, + pool_type: pool_type.clone(), + provider: provider.clone(), + tags: vec!["test".to_string()], + }, + ) + .await + .unwrap(); + + (server_id, datacenter_id) } diff --git a/svc/pkg/cluster/standalone/metrics-publish/Cargo.toml b/svc/pkg/cluster/standalone/metrics-publish/Cargo.toml index bc0073f00..f1f28860c 100644 --- a/svc/pkg/cluster/standalone/metrics-publish/Cargo.toml +++ b/svc/pkg/cluster/standalone/metrics-publish/Cargo.toml @@ -7,17 +7,16 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } rivet-connection = { path = "../../../../../lib/connection" } rivet-health-checks = { path = "../../../../../lib/health-checks" } rivet-metrics = { path = "../../../../../lib/metrics" } -rivet-operation = { path = "../../../../../lib/operation/core" } rivet-runtime = { path = "../../../../../lib/runtime" } tokio = { version = "1.29", features = ["full"] } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } -cluster-datacenter-get = { path = "../../ops/datacenter-get" } +cluster = { path = "../.." } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" @@ -25,5 +24,3 @@ rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" default-features = false [dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } -util-cluster = { package = "rivet-util-cluster", path = "../../util" } diff --git a/svc/pkg/cluster/standalone/metrics-publish/src/lib.rs b/svc/pkg/cluster/standalone/metrics-publish/src/lib.rs index 95b52e2c1..b34fb6e95 100644 --- a/svc/pkg/cluster/standalone/metrics-publish/src/lib.rs +++ b/svc/pkg/cluster/standalone/metrics-publish/src/lib.rs @@ -1,13 +1,15 @@ use std::convert::{TryFrom, TryInto}; -use backend::cluster::PoolType::*; -use proto::backend; -use rivet_operation::prelude::*; -use util_cluster::metrics; +use chirp_workflow::prelude::*; +use cluster::{ + types::{Datacenter, PoolType}, + util::metrics, +}; #[derive(sqlx::FromRow)] struct ServerRow { datacenter_id: Uuid, + pool_type2: Option>, pool_type: i64, is_provisioned: bool, is_installed: bool, @@ -19,7 +21,7 @@ struct ServerRow { struct Server { datacenter_id: Uuid, - pool_type: backend::cluster::PoolType, + pool_type: PoolType, is_provisioned: bool, is_installed: bool, has_nomad_node: bool, @@ -33,7 +35,12 @@ impl TryFrom for Server { fn try_from(value: ServerRow) -> GlobalResult { Ok(Server { datacenter_id: value.datacenter_id, - pool_type: unwrap!(backend::cluster::PoolType::from_i32(value.pool_type as i32)), + // Handle backwards compatibility + pool_type: if let Some(pool_type) = value.pool_type2 { + pool_type.0 + } else { + value.pool_type.try_into()? + }, is_provisioned: value.is_provisioned, is_installed: value.is_installed, has_nomad_node: value.has_nomad_node, @@ -44,30 +51,24 @@ impl TryFrom for Server { } #[tracing::instrument(skip_all)] -pub async fn run_from_env(_ts: i64, pools: rivet_pools::Pools) -> GlobalResult<()> { +pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("cluster-metrics-publish"); let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "cluster-metrics-publish".into(), - std::time::Duration::from_secs(60), + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); + "cluster-metrics-publish", + ) + .await?; let servers = select_servers(&ctx).await?; - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: servers - .iter() - .map(|s| s.datacenter_id.into()) - .collect::>(), - }) - .await?; + let datacenters_res = ctx + .op(cluster::ops::datacenter::get::Input { + datacenter_ids: servers.iter().map(|s| s.datacenter_id).collect::>(), + }) + .await?; for dc in &datacenters_res.datacenters { insert_metrics(dc, &servers)?; @@ -76,12 +77,12 @@ pub async fn run_from_env(_ts: i64, pools: rivet_pools::Pools) -> GlobalResult<( Ok(()) } -async fn select_servers(ctx: &OperationContext<()>) -> GlobalResult> { +async fn select_servers(ctx: &StandaloneCtx) -> GlobalResult> { let servers = sql_fetch_all!( [ctx, ServerRow] " SELECT - datacenter_id, pool_type, + datacenter_id, pool_type, pool_type2, (provider_server_id IS NOT NULL) AS is_provisioned, (install_complete_ts IS NOT NULL) AS is_installed, (nomad_node_id IS NOT NULL) AS has_nomad_node, @@ -102,33 +103,34 @@ async fn select_servers(ctx: &OperationContext<()>) -> GlobalResult> .collect::>>() } -fn insert_metrics(dc: &backend::cluster::Datacenter, servers: &[Server]) -> GlobalResult<()> { - let datacenter_id = unwrap_ref!(dc.datacenter_id).as_uuid(); - let servers_in_dc = servers.iter().filter(|s| s.datacenter_id == datacenter_id); +fn insert_metrics(dc: &Datacenter, servers: &[Server]) -> GlobalResult<()> { + let servers_in_dc = servers + .iter() + .filter(|s| s.datacenter_id == dc.datacenter_id); - let datacenter_id = datacenter_id.to_string(); - let cluster_id = unwrap_ref!(dc.cluster_id).as_uuid().to_string(); + let datacenter_id = dc.datacenter_id.to_string(); + let cluster_id = dc.cluster_id.to_string(); let servers_per_pool = [ ( - Job, + PoolType::Job, servers_in_dc .clone() - .filter(|s| matches!(s.pool_type, Job)) + .filter(|s| matches!(s.pool_type, PoolType::Job)) .collect::>(), ), ( - Gg, + PoolType::Gg, servers_in_dc .clone() - .filter(|s| matches!(s.pool_type, Gg)) + .filter(|s| matches!(s.pool_type, PoolType::Gg)) .collect::>(), ), ( - Ats, + PoolType::Ats, servers_in_dc .clone() - .filter(|s| matches!(s.pool_type, Ats)) + .filter(|s| matches!(s.pool_type, PoolType::Ats)) .collect::>(), ), ]; @@ -174,11 +176,7 @@ fn insert_metrics(dc: &backend::cluster::Datacenter, servers: &[Server]) -> Glob datacenter_id.as_str(), &dc.provider_datacenter_id, &dc.name_id, - match pool_type { - Job => "job", - Gg => "gg", - Ats => "ats", - }, + &pool_type.to_string(), ]; metrics::PROVISIONING_SERVERS @@ -200,7 +198,7 @@ fn insert_metrics(dc: &backend::cluster::Datacenter, servers: &[Server]) -> Glob .with_label_values(&labels) .set(draining_tainted); - if let Job = pool_type { + if let PoolType::Job = pool_type { metrics::NOMAD_SERVERS .with_label_values(&[ &cluster_id, diff --git a/svc/pkg/cluster/standalone/metrics-publish/src/main.rs b/svc/pkg/cluster/standalone/metrics-publish/src/main.rs index 51b151e26..65475901a 100644 --- a/svc/pkg/cluster/standalone/metrics-publish/src/main.rs +++ b/svc/pkg/cluster/standalone/metrics-publish/src/main.rs @@ -1,6 +1,6 @@ use std::time::Duration; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; fn main() -> GlobalResult<()> { rivet_runtime::run(start()).unwrap() @@ -25,7 +25,6 @@ async fn start() -> GlobalResult<()> { loop { interval.tick().await; - let ts = util::timestamp::now(); - cluster_metrics_publish::run_from_env(ts, pools.clone()).await?; + cluster_metrics_publish::run_from_env(pools.clone()).await?; } } diff --git a/svc/pkg/cluster/tests/common.rs b/svc/pkg/cluster/tests/common.rs new file mode 100644 index 000000000..07e57a5b0 --- /dev/null +++ b/svc/pkg/cluster/tests/common.rs @@ -0,0 +1,107 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +pub struct Setup { + pub server_id: Uuid, + pub datacenter_id: Uuid, + pub cluster_id: Uuid, + pub pool_type: cluster::types::PoolType, + pub drain_timeout: u64, +} + +pub struct SetupRes { + pub pools: Vec, + pub provider: cluster::types::Provider, +} + +pub async fn setup(ctx: &TestCtx, opts: Setup) -> SetupRes { + let pools = vec![cluster::types::Pool { + pool_type: opts.pool_type.clone(), + hardware: vec![cluster::types::Hardware { + provider_hardware: cluster::util::test::LINODE_HARDWARE.to_string(), + }], + desired_count: 0, + min_count: 0, + max_count: 0, + drain_timeout: opts.drain_timeout, + }]; + let provider = cluster::types::Provider::Linode; + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": opts.cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": opts.cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id: opts.cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + let mut sub = ctx + .subscribe::(&json!({ + "datacenter_id": opts.datacenter_id, + })) + .await + .unwrap(); + + ctx.tagged_signal( + &json!({ + "cluster_id": opts.cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id: opts.datacenter_id, + name_id: util::faker::ident(), + display_name: util::faker::ident(), + + provider: provider.clone(), + provider_datacenter_id: "us-southeast".to_string(), + provider_api_token: None, + + pools: pools.clone(), + + build_delivery_method: cluster::types::BuildDeliveryMethod::TrafficServer, + prebakes_enabled: false, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + // Write new server to db + sql_execute!( + [ctx] + " + INSERT INTO db_cluster.servers ( + server_id, + datacenter_id, + pool_type2, + create_ts, + + -- Backwards compatibility + pool_type + ) + VALUES ($1, $2, $3, $4, 0) + ", + opts.server_id, + opts.datacenter_id, + serde_json::to_string(&opts.pool_type)?, + util::timestamp::now(), + ) + .await + .unwrap(); + + SetupRes { pools, provider } +} diff --git a/svc/pkg/cluster/tests/create.rs b/svc/pkg/cluster/tests/create.rs new file mode 100644 index 000000000..950a76510 --- /dev/null +++ b/svc/pkg/cluster/tests/create.rs @@ -0,0 +1,38 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +#[workflow_test] +async fn create(ctx: TestCtx) { + let cluster_id = Uuid::new_v4(); + let owner_team_id = Uuid::new_v4(); + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: Some(owner_team_id), + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + let res = ctx + .op(cluster::ops::get::Input { + cluster_ids: vec![cluster_id], + }) + .await + .unwrap(); + assert!(!res.clusters.is_empty(), "cluster not found"); +} diff --git a/svc/pkg/cluster/tests/datacenter_create.rs b/svc/pkg/cluster/tests/datacenter_create.rs new file mode 100644 index 000000000..292eca6d1 --- /dev/null +++ b/svc/pkg/cluster/tests/datacenter_create.rs @@ -0,0 +1,69 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +#[workflow_test] +async fn datacenter_create(ctx: TestCtx) { + let datacenter_id = Uuid::new_v4(); + let cluster_id = Uuid::new_v4(); + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + ctx.tagged_signal( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id, + name_id: util::faker::ident(), + display_name: util::faker::ident(), + + provider: cluster::types::Provider::Linode, + provider_datacenter_id: "us-southeast".to_string(), + provider_api_token: None, + + pools: Vec::new(), + + build_delivery_method: cluster::types::BuildDeliveryMethod::TrafficServer, + prebakes_enabled: false, + }, + ) + .await + .unwrap(); + + // Check if tls record exists + let (exists,) = sql_fetch_one!( + [ctx, (bool,)] + " + SELECT EXISTS ( + SELECT 1 + FROM db_cluster.datacenter_tls + WHERE datacenter_id = $1 + ) + ", + datacenter_id, + ) + .await + .unwrap(); + + assert!(exists, "no tls record"); +} diff --git a/svc/pkg/cluster/tests/get.rs b/svc/pkg/cluster/tests/get.rs new file mode 100644 index 000000000..4a5b85d34 --- /dev/null +++ b/svc/pkg/cluster/tests/get.rs @@ -0,0 +1,39 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +#[workflow_test] +async fn get(ctx: TestCtx) { + let cluster_id = Uuid::new_v4(); + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + let res = ctx + .op(cluster::ops::get::Input { + cluster_ids: vec![cluster_id], + }) + .await + .unwrap(); + let cluster = res.clusters.first().expect("cluster not found"); + + assert_eq!(cluster_id, cluster.cluster_id); +} diff --git a/svc/pkg/cluster/tests/get_for_game.rs b/svc/pkg/cluster/tests/get_for_game.rs new file mode 100644 index 000000000..e2dc0150f --- /dev/null +++ b/svc/pkg/cluster/tests/get_for_game.rs @@ -0,0 +1,49 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +#[workflow_test] +async fn get_for_game(ctx: TestCtx) { + let cluster_id = Uuid::new_v4(); + let game_id = Uuid::new_v4(); + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + ctx.tagged_signal( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::GameLink { game_id }, + ) + .await + .unwrap(); + + let games_res = ctx + .op(cluster::ops::get_for_game::Input { + game_ids: vec![game_id], + }) + .await + .unwrap(); + let game = games_res.games.first().unwrap(); + + assert_eq!(cluster_id, game.cluster_id); +} diff --git a/svc/pkg/cluster/tests/list.rs b/svc/pkg/cluster/tests/list.rs new file mode 100644 index 000000000..26488afec --- /dev/null +++ b/svc/pkg/cluster/tests/list.rs @@ -0,0 +1,37 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +#[workflow_test] +async fn list_single_cluster(ctx: TestCtx) { + let cluster_id = Uuid::new_v4(); + + let mut sub = ctx + .subscribe::(&json!({ + "cluster_id": cluster_id, + })) + .await + .unwrap(); + + ctx.dispatch_tagged_workflow( + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) + .await + .unwrap(); + + sub.next().await.unwrap(); + + let res = ctx.op(cluster::ops::list::Input {}).await.unwrap(); + + // The cluster should be in the list of all clusters + res.cluster_ids + .into_iter() + .find(|id| id == &cluster_id) + .unwrap(); +} diff --git a/svc/pkg/cluster/worker/tests/server_provision.rs b/svc/pkg/cluster/tests/server_provision.rs similarity index 61% rename from svc/pkg/cluster/worker/tests/server_provision.rs rename to svc/pkg/cluster/tests/server_provision.rs index 3140b1caa..d302c0157 100644 --- a/svc/pkg/cluster/worker/tests/server_provision.rs +++ b/svc/pkg/cluster/tests/server_provision.rs @@ -1,10 +1,10 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; +use serde_json::json; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_provision(ctx: TestCtx) { if !util::feature::server_provision() { return; @@ -20,19 +20,22 @@ async fn server_provision(ctx: TestCtx) { server_id, datacenter_id, cluster_id, - pool_type: backend::cluster::PoolType::Ats, + pool_type: cluster::types::PoolType::Ats, drain_timeout: 0, }, ) .await; - msg!([ctx] cluster::msg::server_provision(server_id) { - datacenter_id: Some(datacenter_id.into()), - server_id: Some(server_id.into()), - pool_type: dc.pools.first().unwrap().pool_type, - provider: dc.provider as i32, - tags: vec!["test".to_string()], - }) + ctx.tagged_signal( + &json!({ + "datacenter_id": datacenter_id, + }), + cluster::workflows::datacenter::ServerCreate { + server_id, + pool_type: dc.pools.first().unwrap().pool_type.clone(), + tags: vec!["test".to_string()], + }, + ) .await .unwrap(); @@ -62,10 +65,12 @@ async fn server_provision(ctx: TestCtx) { } // Clean up afterwards so we don't litter - msg!([ctx] @wait cluster::msg::server_destroy(server_id) { - server_id: Some(server_id.into()), - force: false, - }) + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + cluster::workflows::server::Destroy {}, + ) .await .unwrap(); } diff --git a/svc/pkg/cluster/ops/datacenter-get/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_get.rs similarity index 93% rename from svc/pkg/cluster/ops/datacenter-get/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_get.rs index a2669d93b..9dc60b02d 100644 --- a/svc/pkg/cluster/ops/datacenter-get/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_get.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); diff --git a/svc/pkg/cluster/ops/datacenter-list/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_list.rs similarity index 93% rename from svc/pkg/cluster/ops/datacenter-list/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_list.rs index ff707b2a2..7cbb9747d 100644 --- a/svc/pkg/cluster/ops/datacenter-list/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_list.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); diff --git a/svc/pkg/cluster/ops/datacenter-location-get/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_location_get.rs similarity index 95% rename from svc/pkg/cluster/ops/datacenter-location-get/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_location_get.rs index cb9c2409f..09fff894d 100644 --- a/svc/pkg/cluster/ops/datacenter-location-get/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_location_get.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn basic(ctx: TestCtx) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); diff --git a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_resolve_for_name_id.rs similarity index 93% rename from svc/pkg/cluster/ops/datacenter-resolve-for-name-id/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_resolve_for_name_id.rs index ab185ce60..40616a68e 100644 --- a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_resolve_for_name_id.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); diff --git a/svc/pkg/cluster/worker/tests/datacenter_scale.rs b/svc/pkg/cluster/testsTMP/datacenter_scale.rs similarity index 69% rename from svc/pkg/cluster/worker/tests/datacenter_scale.rs rename to svc/pkg/cluster/testsTMP/datacenter_scale.rs index 4d1c03273..3d2d6d7b3 100644 --- a/svc/pkg/cluster/worker/tests/datacenter_scale.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_scale.rs @@ -1,6 +1,6 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn datacenter_scale(_ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/ops/datacenter-tls-get/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_tls_get.rs similarity index 95% rename from svc/pkg/cluster/ops/datacenter-tls-get/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_tls_get.rs index 4d81a6da4..b1f3132c6 100644 --- a/svc/pkg/cluster/ops/datacenter-tls-get/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_tls_get.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { if !util::feature::dns() { return; diff --git a/svc/pkg/cluster/worker/tests/datacenter_tls_issue.rs b/svc/pkg/cluster/testsTMP/datacenter_tls_issue.rs similarity index 94% rename from svc/pkg/cluster/worker/tests/datacenter_tls_issue.rs rename to svc/pkg/cluster/testsTMP/datacenter_tls_issue.rs index 23b7ca165..94bbaf6e9 100644 --- a/svc/pkg/cluster/worker/tests/datacenter_tls_issue.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_tls_issue.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn datacenter_tls_issue(ctx: TestCtx) { if !util::feature::dns() { return; diff --git a/svc/pkg/cluster/ops/datacenter-topology-get/tests/integration.rs b/svc/pkg/cluster/testsTMP/datacenter_topology_get.rs similarity index 72% rename from svc/pkg/cluster/ops/datacenter-topology-get/tests/integration.rs rename to svc/pkg/cluster/testsTMP/datacenter_topology_get.rs index c31c959da..b33d2b763 100644 --- a/svc/pkg/cluster/ops/datacenter-topology-get/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_topology_get.rs @@ -1,6 +1,6 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { op!([ctx] cluster_datacenter_topology_get { datacenter_ids: vec![], diff --git a/svc/pkg/cluster/worker/tests/datacenter_update.rs b/svc/pkg/cluster/testsTMP/datacenter_update.rs similarity index 95% rename from svc/pkg/cluster/worker/tests/datacenter_update.rs rename to svc/pkg/cluster/testsTMP/datacenter_update.rs index 19aa9d8c4..b33a0a427 100644 --- a/svc/pkg/cluster/worker/tests/datacenter_update.rs +++ b/svc/pkg/cluster/testsTMP/datacenter_update.rs @@ -1,7 +1,6 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn datacenter_update(ctx: TestCtx) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); diff --git a/svc/pkg/cluster/worker/tests/nomad_node_drain_complete.rs b/svc/pkg/cluster/testsTMP/nomad_node_drain_complete.rs similarity index 91% rename from svc/pkg/cluster/worker/tests/nomad_node_drain_complete.rs rename to svc/pkg/cluster/testsTMP/nomad_node_drain_complete.rs index ee3a7a346..2eb887562 100644 --- a/svc/pkg/cluster/worker/tests/nomad_node_drain_complete.rs +++ b/svc/pkg/cluster/testsTMP/nomad_node_drain_complete.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn nomad_node_drain_complete(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/worker/tests/nomad_node_registered.rs b/svc/pkg/cluster/testsTMP/nomad_node_registered.rs similarity index 91% rename from svc/pkg/cluster/worker/tests/nomad_node_registered.rs rename to svc/pkg/cluster/testsTMP/nomad_node_registered.rs index 70f07a8d1..f93971ea4 100644 --- a/svc/pkg/cluster/worker/tests/nomad_node_registered.rs +++ b/svc/pkg/cluster/testsTMP/nomad_node_registered.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn nomad_node_registered(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/ops/resolve-for-name-id/tests/integration.rs b/svc/pkg/cluster/testsTMP/resolve_for_name_id.rs similarity index 92% rename from svc/pkg/cluster/ops/resolve-for-name-id/tests/integration.rs rename to svc/pkg/cluster/testsTMP/resolve_for_name_id.rs index 27ef5816c..5d38ffde8 100644 --- a/svc/pkg/cluster/ops/resolve-for-name-id/tests/integration.rs +++ b/svc/pkg/cluster/testsTMP/resolve_for_name_id.rs @@ -1,7 +1,7 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; use proto::backend::pkg::*; -#[worker_test] +#[workflow_test] async fn empty(ctx: TestCtx) { let cluster_id = Uuid::new_v4(); let name_id = util::faker::ident(); diff --git a/svc/pkg/cluster/worker/tests/server_destroy.rs b/svc/pkg/cluster/testsTMP/server_destroy.rs similarity index 93% rename from svc/pkg/cluster/worker/tests/server_destroy.rs rename to svc/pkg/cluster/testsTMP/server_destroy.rs index 36e1aaf08..d17652495 100644 --- a/svc/pkg/cluster/worker/tests/server_destroy.rs +++ b/svc/pkg/cluster/testsTMP/server_destroy.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_destroy(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/testsTMP/server_destroy_with_filter.rs b/svc/pkg/cluster/testsTMP/server_destroy_with_filter.rs new file mode 100644 index 000000000..d37cae309 --- /dev/null +++ b/svc/pkg/cluster/testsTMP/server_destroy_with_filter.rs @@ -0,0 +1,6 @@ +use chirp_workflow::prelude::*; + +#[workflow_test] +async fn basic(ctx: TestCtx) { + // TODO: +} diff --git a/svc/pkg/cluster/worker/tests/server_dns_create.rs b/svc/pkg/cluster/testsTMP/server_dns_create.rs similarity index 93% rename from svc/pkg/cluster/worker/tests/server_dns_create.rs rename to svc/pkg/cluster/testsTMP/server_dns_create.rs index 078913147..bff334ca1 100644 --- a/svc/pkg/cluster/worker/tests/server_dns_create.rs +++ b/svc/pkg/cluster/testsTMP/server_dns_create.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_dns_create(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/worker/tests/server_dns_delete.rs b/svc/pkg/cluster/testsTMP/server_dns_delete.rs similarity index 93% rename from svc/pkg/cluster/worker/tests/server_dns_delete.rs rename to svc/pkg/cluster/testsTMP/server_dns_delete.rs index 0e56af1ad..cccd80f7a 100644 --- a/svc/pkg/cluster/worker/tests/server_dns_delete.rs +++ b/svc/pkg/cluster/testsTMP/server_dns_delete.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_dns_delete(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/worker/tests/server_drain.rs b/svc/pkg/cluster/testsTMP/server_drain.rs similarity index 96% rename from svc/pkg/cluster/worker/tests/server_drain.rs rename to svc/pkg/cluster/testsTMP/server_drain.rs index d6c1f7175..bb1ebe1cf 100644 --- a/svc/pkg/cluster/worker/tests/server_drain.rs +++ b/svc/pkg/cluster/testsTMP/server_drain.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_drain(ctx: TestCtx) { if !util::feature::server_provision() { return; @@ -51,7 +50,7 @@ async fn server_drain(ctx: TestCtx) { .unwrap(); } -#[worker_test] +#[workflow_test] async fn gg_server_drain(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/testsTMP/server_get.rs b/svc/pkg/cluster/testsTMP/server_get.rs new file mode 100644 index 000000000..46f5134b9 --- /dev/null +++ b/svc/pkg/cluster/testsTMP/server_get.rs @@ -0,0 +1,6 @@ +use chirp_workflow::prelude::*; + +#[workflow_test] +async fn empty(_ctx: TestCtx) { + // TODO: +} diff --git a/svc/pkg/cluster/worker/tests/server_install.rs b/svc/pkg/cluster/testsTMP/server_install.rs similarity index 94% rename from svc/pkg/cluster/worker/tests/server_install.rs rename to svc/pkg/cluster/testsTMP/server_install.rs index 30d22cf9b..b289df1e7 100644 --- a/svc/pkg/cluster/worker/tests/server_install.rs +++ b/svc/pkg/cluster/testsTMP/server_install.rs @@ -1,10 +1,9 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn server_install(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/worker/tests/server_install_complete.rs b/svc/pkg/cluster/testsTMP/server_install_complete.rs similarity index 75% rename from svc/pkg/cluster/worker/tests/server_install_complete.rs rename to svc/pkg/cluster/testsTMP/server_install_complete.rs index 4807150c4..3dca37e15 100644 --- a/svc/pkg/cluster/worker/tests/server_install_complete.rs +++ b/svc/pkg/cluster/testsTMP/server_install_complete.rs @@ -1,6 +1,6 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn server_install_complete(_ctx: TestCtx) { // msg!([ctx] cluster::msg::server_install_complete() { diff --git a/svc/pkg/cluster/testsTMP/server_list.rs b/svc/pkg/cluster/testsTMP/server_list.rs new file mode 100644 index 000000000..46f5134b9 --- /dev/null +++ b/svc/pkg/cluster/testsTMP/server_list.rs @@ -0,0 +1,6 @@ +use chirp_workflow::prelude::*; + +#[workflow_test] +async fn empty(_ctx: TestCtx) { + // TODO: +} diff --git a/svc/pkg/cluster/testsTMP/server_resolve_for_ip.rs b/svc/pkg/cluster/testsTMP/server_resolve_for_ip.rs new file mode 100644 index 000000000..46f5134b9 --- /dev/null +++ b/svc/pkg/cluster/testsTMP/server_resolve_for_ip.rs @@ -0,0 +1,6 @@ +use chirp_workflow::prelude::*; + +#[workflow_test] +async fn empty(_ctx: TestCtx) { + // TODO: +} diff --git a/svc/pkg/cluster/worker/tests/server_taint.rs b/svc/pkg/cluster/testsTMP/server_taint.rs similarity index 97% rename from svc/pkg/cluster/worker/tests/server_taint.rs rename to svc/pkg/cluster/testsTMP/server_taint.rs index 3ff12fb6f..ef2ff99d1 100644 --- a/svc/pkg/cluster/worker/tests/server_taint.rs +++ b/svc/pkg/cluster/testsTMP/server_taint.rs @@ -1,12 +1,11 @@ use std::time::Duration; -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use chirp_workflow::prelude::*; mod common; use common::{setup, Setup}; -#[worker_test] +#[workflow_test] async fn datacenter_taint(ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/worker/tests/server_undrain.rs b/svc/pkg/cluster/testsTMP/server_undrain.rs similarity index 90% rename from svc/pkg/cluster/worker/tests/server_undrain.rs rename to svc/pkg/cluster/testsTMP/server_undrain.rs index 93f5c6069..ca2121260 100644 --- a/svc/pkg/cluster/worker/tests/server_undrain.rs +++ b/svc/pkg/cluster/testsTMP/server_undrain.rs @@ -1,6 +1,6 @@ -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; -#[worker_test] +#[workflow_test] async fn server_undrain(_ctx: TestCtx) { if !util::feature::server_provision() { return; diff --git a/svc/pkg/cluster/util/Cargo.toml b/svc/pkg/cluster/util/Cargo.toml deleted file mode 100644 index b1065186b..000000000 --- a/svc/pkg/cluster/util/Cargo.toml +++ /dev/null @@ -1,18 +0,0 @@ -[package] -name = "rivet-util-cluster" -version = "0.1.0" -edition = "2021" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -lazy_static = "1.4" -rivet-metrics = { path = "../../../../lib/metrics" } -rivet-util = { path = "../../../../lib/util/core" } -types = { path = "../../../../lib/types/core" } -uuid = { version = "1", features = ["v4", "serde"] } - -[build-dependencies] -merkle_hash = "3.6" -hex = "0.4" -tokio = { version = "1.29", features = ["full"] } diff --git a/svc/pkg/cluster/worker/Cargo.toml b/svc/pkg/cluster/worker/Cargo.toml deleted file mode 100644 index 7fefd6abd..000000000 --- a/svc/pkg/cluster/worker/Cargo.toml +++ /dev/null @@ -1,51 +0,0 @@ -[package] -name = "cluster-worker" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -acme-lib = "0.9" -anyhow = "1.0" -chirp-client = { path = "../../../../lib/chirp/client" } -chirp-worker = { path = "../../../../lib/chirp/worker" } -chrono = "0.4" -cloudflare = "0.10.1" -http = "0.2" -include_dir = "0.7.3" -indoc = "1.0" -lazy_static = "1.4" -maplit = "1.0" -nomad-util = { path = "../../../../lib/nomad-util" } -openssl = "0.10.63" -rivet-convert = { path = "../../../../lib/convert" } -rivet-health-checks = { path = "../../../../lib/health-checks" } -rivet-metrics = { path = "../../../../lib/metrics" } -rivet-runtime = { path = "../../../../lib/runtime" } -s3-util = { path = "../../../../lib/s3-util" } -serde_yaml = "0.9" -ssh2 = "0.9.4" -thiserror = "1.0" -trust-dns-resolver = { version = "0.23.2", features = ["dns-over-native-tls"] } -util-cluster = { package = "rivet-util-cluster", path = "../util" } - -cluster-datacenter-get = { path = "../ops/datacenter-get" } -cluster-datacenter-list = { path = "../ops/datacenter-list" } -cluster-datacenter-topology-get = { path = "../ops/datacenter-topology-get" } -linode-instance-type-get = { path = "../../linode/ops/instance-type-get" } -linode-server-destroy = { path = "../../linode/ops/server-destroy" } -linode-server-provision = { path = "../../linode/ops/server-provision" } -token-create = { path = "../../token/ops/create" } - -[dependencies.nomad_client] -git = "https://github.com/rivet-gg/nomad-client" -rev = "abb66bf0c30c7ff5b0c695dae952481c33e538b5" # pragma: allowlist secret - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../lib/chirp/worker" } diff --git a/svc/pkg/cluster/worker/src/lib.rs b/svc/pkg/cluster/worker/src/lib.rs deleted file mode 100644 index beb1874f4..000000000 --- a/svc/pkg/cluster/worker/src/lib.rs +++ /dev/null @@ -1,2 +0,0 @@ -pub mod util; -pub mod workers; diff --git a/svc/pkg/cluster/worker/src/util.rs b/svc/pkg/cluster/worker/src/util.rs deleted file mode 100644 index 9cc49fea5..000000000 --- a/svc/pkg/cluster/worker/src/util.rs +++ /dev/null @@ -1,6 +0,0 @@ -#[derive(thiserror::Error, Debug)] -#[error("cloudflare: {source}")] -pub struct CloudflareError { - #[from] - source: anyhow::Error, -} diff --git a/svc/pkg/cluster/worker/src/workers/create.rs b/svc/pkg/cluster/worker/src/workers/create.rs deleted file mode 100644 index 8f6aee608..000000000 --- a/svc/pkg/cluster/worker/src/workers/create.rs +++ /dev/null @@ -1,33 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker(name = "cluster-create")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - let cluster_id = unwrap_ref!(ctx.cluster_id).as_uuid(); - let owner_team_id = ctx.owner_team_id.map(|id| id.as_uuid()); - - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.clusters ( - cluster_id, - name_id, - owner_team_id, - create_ts - ) - VALUES ($1, $2, $3, $4) - ", - cluster_id, - &ctx.name_id, - owner_team_id, - util::timestamp::now(), - ) - .await?; - - msg!([ctx] cluster::msg::create_complete(cluster_id) { - cluster_id: ctx.cluster_id - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/datacenter_create.rs b/svc/pkg/cluster/worker/src/workers/datacenter_create.rs deleted file mode 100644 index d04018c80..000000000 --- a/svc/pkg/cluster/worker/src/workers/datacenter_create.rs +++ /dev/null @@ -1,98 +0,0 @@ -use chirp_worker::prelude::*; -use futures_util::FutureExt; -use proto::backend::{self, pkg::*}; - -#[worker(name = "cluster-datacenter-create")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let cluster_id = unwrap_ref!(ctx.cluster_id).as_uuid(); - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - - let mut pools = ctx.pools.clone(); - - // Constrain the desired count - for pool in &mut pools { - pool.desired_count = pool.desired_count.max(pool.min_count).min(pool.max_count); - } - - // Copy pools config to write to db - let pools = cluster::msg::datacenter_create::Pools { pools }; - - let mut pools_buf = Vec::with_capacity(pools.encoded_len()); - pools.encode(&mut pools_buf)?; - - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - let ctx = ctx.clone(); - let pools_buf = pools_buf.clone(); - - async move { - sql_execute!( - [ctx, @tx tx] - " - INSERT INTO db_cluster.datacenters ( - datacenter_id, - cluster_id, - name_id, - display_name, - provider, - provider_datacenter_id, - provider_api_token, - pools, - build_delivery_method, - prebakes_enabled, - create_ts - ) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11) - ", - datacenter_id, - cluster_id, - &ctx.name_id, - &ctx.display_name, - ctx.provider as i64, - &ctx.provider_datacenter_id, - &ctx.provider_api_token, - pools_buf, - ctx.build_delivery_method as i64, - ctx.prebakes_enabled, - util::timestamp::now(), - ) - .await?; - - // Insert TLS record - sql_execute!( - [ctx, @tx tx] - " - INSERT INTO db_cluster.datacenter_tls ( - datacenter_id, - state, - expire_ts - ) - VALUES ($1, $2, 0) - ", - datacenter_id, - backend::cluster::TlsState::Creating as i64, - ) - .await?; - - Ok(()) - } - .boxed() - }) - .await?; - - // Start TLS issuing process - msg!([ctx] cluster::msg::datacenter_tls_issue(datacenter_id) { - datacenter_id: ctx.datacenter_id, - renew: false, - }) - .await?; - - // Scale servers - msg!([ctx] cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: ctx.datacenter_id, - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/datacenter_update.rs b/svc/pkg/cluster/worker/src/workers/datacenter_update.rs deleted file mode 100644 index e29663bd3..000000000 --- a/svc/pkg/cluster/worker/src/workers/datacenter_update.rs +++ /dev/null @@ -1,81 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker(name = "cluster-datacenter-update")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!( - datacenter_res.datacenters.first(), - "datacenter does not exist" - ); - - // Update pools config - let mut new_pools = cluster::msg::datacenter_create::Pools { - pools: datacenter.pools.clone(), - }; - for pool in &ctx.pools { - let current_pool = unwrap!( - new_pools - .pools - .iter_mut() - .find(|p| p.pool_type == pool.pool_type), - "attempting to update pool that doesn't exist in current config" - ); - - // Update pool config - if !pool.hardware.is_empty() { - current_pool.hardware.clone_from(&pool.hardware); - } - if let Some(desired_count) = pool.desired_count { - current_pool.desired_count = desired_count; - } - if let Some(min_count) = pool.min_count { - current_pool.min_count = min_count; - } - if let Some(max_count) = pool.max_count { - current_pool.max_count = max_count; - } - if let Some(drain_timeout) = pool.drain_timeout { - current_pool.drain_timeout = drain_timeout; - } - } - - // Encode config - let mut pools_buf = Vec::with_capacity(new_pools.encoded_len()); - new_pools.encode(&mut pools_buf)?; - - // Update pools - sql_execute!( - [ctx] - " - UPDATE db_cluster.datacenters - SET - pools = $2, - prebakes_enabled = coalesce($3, prebakes_enabled) - WHERE datacenter_id = $1 - ", - datacenter_id, - pools_buf, - ctx.prebakes_enabled, - ) - .await?; - - // Purge cache - ctx.cache() - .purge("cluster.datacenters", [datacenter_id]) - .await?; - - msg!([ctx] cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: ctx.datacenter_id, - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/game_link.rs b/svc/pkg/cluster/worker/src/workers/game_link.rs deleted file mode 100644 index 64f241cbb..000000000 --- a/svc/pkg/cluster/worker/src/workers/game_link.rs +++ /dev/null @@ -1,30 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker(name = "cluster-game-link")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - let game_id = unwrap_ref!(ctx.game_id).as_uuid(); - let cluster_id = unwrap_ref!(ctx.cluster_id).as_uuid(); - - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.games ( - game_id, - cluster_id - ) - VALUES ($1, $2) - ", - game_id, - cluster_id, - ) - .await?; - - msg!([ctx] cluster::msg::game_link_complete(game_id, cluster_id) { - game_id: ctx.game_id, - cluster_id: ctx.cluster_id, - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/mod.rs b/svc/pkg/cluster/worker/src/workers/mod.rs deleted file mode 100644 index a943ba64d..000000000 --- a/svc/pkg/cluster/worker/src/workers/mod.rs +++ /dev/null @@ -1,37 +0,0 @@ -pub mod create; -pub mod datacenter_create; -pub mod datacenter_scale; -pub mod datacenter_tls_issue; -pub mod datacenter_update; -pub mod game_link; -pub mod nomad_node_drain_complete; -pub mod nomad_node_registered; -pub mod server_destroy; -pub mod server_dns_create; -pub mod server_dns_delete; -pub mod server_drain; -pub mod server_install; -pub mod server_install_complete; -pub mod server_provision; -pub mod server_taint; -pub mod server_undrain; - -chirp_worker::workers![ - server_taint, - create, - datacenter_create, - datacenter_scale, - datacenter_tls_issue, - datacenter_update, - game_link, - nomad_node_drain_complete, - nomad_node_registered, - server_destroy, - server_dns_create, - server_dns_delete, - server_drain, - server_install_complete, - server_install, - server_provision, - server_undrain, -]; diff --git a/svc/pkg/cluster/worker/src/workers/nomad_node_drain_complete.rs b/svc/pkg/cluster/worker/src/workers/nomad_node_drain_complete.rs deleted file mode 100644 index 681009a5c..000000000 --- a/svc/pkg/cluster/worker/src/workers/nomad_node_drain_complete.rs +++ /dev/null @@ -1,30 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker(name = "cluster-nomad-node-drain-complete")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - // Set as completed draining. Will be destroyed by `cluster-datacenter-scale` - let (datacenter_id,) = sql_fetch_one!( - [ctx, (Uuid,)] - " - UPDATE db_cluster.servers - SET drain_complete_ts = $2 - WHERE server_id = $1 - RETURNING datacenter_id - ", - &server_id, - util::timestamp::now(), - ) - .await?; - - msg!([ctx] cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: Some(datacenter_id.into()), - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/nomad_node_registered.rs b/svc/pkg/cluster/worker/src/workers/nomad_node_registered.rs deleted file mode 100644 index bd09e974b..000000000 --- a/svc/pkg/cluster/worker/src/workers/nomad_node_registered.rs +++ /dev/null @@ -1,75 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; -use util_cluster::metrics; - -#[worker(name = "cluster-nomad-node-registered")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - let nomad_join_ts = util::timestamp::now(); - - let (datacenter_id, old_nomad_node_id, install_complete_ts) = sql_fetch_one!( - [ctx, (Uuid, Option, Option)] - " - UPDATE db_cluster.servers - SET - nomad_node_id = $2, - nomad_join_ts = $3 - WHERE - server_id = $1 - RETURNING datacenter_id, nomad_node_id, install_complete_ts - ", - &server_id, - &ctx.node_id, - nomad_join_ts, - ) - .await?; - - if let Some(old_nomad_node_id) = old_nomad_node_id { - tracing::warn!(%old_nomad_node_id, "nomad node id was already set"); - } - - // Scale to get rid of tainted servers - msg!([ctx] cluster::msg::datacenter_scale(datacenter_id) { - datacenter_id: Some(datacenter_id.into()), - }) - .await?; - - // Insert metrics - if let Some(install_complete_ts) = install_complete_ts { - insert_metrics(ctx, datacenter_id, nomad_join_ts, install_complete_ts).await?; - } else { - tracing::warn!("missing install_complete_ts for nomad-node-registered"); - } - - Ok(()) -} - -async fn insert_metrics( - ctx: &OperationContext, - datacenter_id: Uuid, - nomad_join_ts: i64, - install_complete_ts: i64, -) -> GlobalResult<()> { - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let dc = unwrap!(datacenters_res.datacenters.first()); - - let datacenter_id = datacenter_id.to_string(); - let cluster_id = unwrap_ref!(dc.cluster_id).as_uuid().to_string(); - let dt = (nomad_join_ts - install_complete_ts) as f64 / 1000.0; - - metrics::NOMAD_JOIN_DURATION - .with_label_values(&[ - cluster_id.as_str(), - datacenter_id.as_str(), - &dc.provider_datacenter_id, - &dc.name_id, - ]) - .observe(dt); - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_destroy.rs b/svc/pkg/cluster/worker/src/workers/server_destroy.rs deleted file mode 100644 index fabf823f4..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_destroy.rs +++ /dev/null @@ -1,85 +0,0 @@ -use chirp_worker::prelude::*; -use futures_util::FutureExt; -use proto::backend::{self, pkg::*}; - -#[derive(sqlx::FromRow)] -struct Server { - datacenter_id: Uuid, - pool_type: i64, - provider_server_id: Option, -} - -#[worker(name = "cluster-server-destroy")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - // TODO: RVTEE-75 - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| inner(ctx.clone(), tx).boxed()).await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - _tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - // NOTE: We do not check if `cloud_destroy_ts` is set because of the destroy fn in - // `cluster-server-provision` - let server = sql_fetch_one!( - [ctx, Server] - " - SELECT datacenter_id, pool_type, provider_server_id - FROM db_cluster.servers - WHERE server_id = $1 - ", - &server_id, - util::timestamp::now(), - ) - .await?; - if server.provider_server_id.is_none() && !ctx.force { - // Check for stale message - if ctx.req_dt() > util::duration::hours(1) { - tracing::warn!("discarding stale message"); - return Ok(()); - } - - bail!("server is not completely provisioned yet, retrying"); - } - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![server.datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - let provider = unwrap!(backend::cluster::Provider::from_i32(datacenter.provider)); - - match provider { - backend::cluster::Provider::Linode => { - tracing::info!(?server_id, "destroying linode server"); - - op!([ctx] linode_server_destroy { - server_id: ctx.server_id, - datacenter_id: Some(server.datacenter_id.into()), - }) - .await?; - } - } - - // Delete DNS record - let pool_type = unwrap!(backend::cluster::PoolType::from_i32( - server.pool_type as i32 - )); - if let backend::cluster::PoolType::Gg = pool_type { - msg!([ctx] cluster::msg::server_dns_delete(server_id) { - server_id: ctx.server_id, - }) - .await?; - } - - msg!([ctx] cluster::msg::server_destroy_complete(server_id) { - server_id: ctx.server_id, - }) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_dns_create.rs b/svc/pkg/cluster/worker/src/workers/server_dns_create.rs deleted file mode 100644 index 5fef14165..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_dns_create.rs +++ /dev/null @@ -1,160 +0,0 @@ -use std::{net::IpAddr, sync::Arc}; - -use chirp_worker::prelude::*; -use cloudflare::{endpoints as cf, framework as cf_framework, framework::async_api::ApiClient}; -use futures_util::FutureExt; -use proto::backend::pkg::*; - -use crate::util::CloudflareError; - -#[derive(sqlx::FromRow)] -struct Server { - datacenter_id: Uuid, - public_ip: IpAddr, - is_destroying_or_draining: bool, -} - -#[worker(name = "cluster-server-dns-create")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let cf_token = util::env::read_secret(&["cloudflare", "terraform", "auth_token"]).await?; - // Create cloudflare HTTP client - let client = Arc::new( - cf_framework::async_api::Client::new( - cf_framework::auth::Credentials::UserAuthToken { token: cf_token }, - Default::default(), - cf_framework::Environment::Production, - ) - .map_err(CloudflareError::from)?, - ); - - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - inner(ctx.clone(), tx, client.clone()).boxed() - }) - .await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - client: Arc, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - // Lock row - sql_execute!( - [ctx, @tx tx] - " - SELECT 1 FROM db_cluster.servers_cloudflare - WHERE - server_id = $1 AND - destroy_ts IS NULL - FOR UPDATE - ", - server_id, - ) - .await?; - - let server = sql_fetch_one!( - [ctx, Server, @tx tx] - " - SELECT - datacenter_id, - public_ip, - (cloud_destroy_ts IS NOT NULL OR drain_ts IS NOT NULL) AS is_destroying_or_draining - FROM db_cluster.servers - WHERE server_id = $1 - ", - server_id, - ) - .await?; - - if server.is_destroying_or_draining { - tracing::info!("server marked for deletion/drain, not creating dns record"); - return Ok(()); - } - - let zone_id = unwrap!(util::env::cloudflare::zone::job::id(), "dns not configured"); - let public_ip = match server.public_ip { - IpAddr::V4(ip) => ip, - IpAddr::V6(_) => bail!("unexpected ipv6 public ip"), - }; - - let record_name = format!( - "*.lobby.{}.{}", - server.datacenter_id, - unwrap!(util::env::domain_job()), - ); - let create_record_res = client - .request(&cf::dns::CreateDnsRecord { - zone_identifier: zone_id, - params: cf::dns::CreateDnsRecordParams { - name: &record_name, - content: cf::dns::DnsContent::A { content: public_ip }, - proxied: Some(false), - ttl: Some(60), - priority: None, - }, - }) - .await?; - - tracing::info!(record_id=%create_record_res.result.id, "created dns record"); - - // Save record id for deletion - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.servers_cloudflare - SET dns_record_id = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - create_record_res.result.id, - ) - .await?; - - // This is solely for compatibility with Discord activities. Their docs say they support parameter - // mapping but it does not work - // https://discord.com/developers/docs/activities/development-guides#prefixtarget-formatting-rules - let secondary_record_name = format!( - "lobby.{}.{}", - server.datacenter_id, - unwrap!(util::env::domain_job()), - ); - let create_secondary_record_res = client - .request(&cf::dns::CreateDnsRecord { - zone_identifier: zone_id, - params: cf::dns::CreateDnsRecordParams { - name: &secondary_record_name, - content: cf::dns::DnsContent::A { content: public_ip }, - proxied: Some(false), - ttl: Some(60), - priority: None, - }, - }) - .await?; - - tracing::info!(record_id=%create_secondary_record_res.result.id, "created secondary dns record"); - - // Save record id for deletion - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.servers_cloudflare - SET secondary_dns_record_id = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - create_secondary_record_res.result.id, - ) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_dns_delete.rs b/svc/pkg/cluster/worker/src/workers/server_dns_delete.rs deleted file mode 100644 index 6f010f363..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_dns_delete.rs +++ /dev/null @@ -1,136 +0,0 @@ -use std::sync::Arc; - -use chirp_worker::prelude::*; -use cloudflare::{endpoints as cf, framework as cf_framework, framework::async_api::ApiClient}; -use futures_util::FutureExt; -use proto::backend::pkg::*; - -use crate::util::CloudflareError; - -#[derive(sqlx::FromRow)] -struct DnsRecordRow { - dns_record_id: Option, - secondary_dns_record_id: Option, -} - -#[worker(name = "cluster-server-dns-delete")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let cf_token = util::env::read_secret(&["cloudflare", "terraform", "auth_token"]).await?; - // Create cloudflare HTTP client - let client = Arc::new( - cf_framework::async_api::Client::new( - cf_framework::auth::Credentials::UserAuthToken { token: cf_token }, - Default::default(), - cf_framework::Environment::Production, - ) - .map_err(CloudflareError::from)?, - ); - - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - inner(ctx.clone(), tx, client.clone()).boxed() - }) - .await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - client: Arc, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - let row = sql_fetch_optional!( - [ctx, DnsRecordRow, @tx tx] - " - SELECT dns_record_id, secondary_dns_record_id - FROM db_cluster.servers_cloudflare - WHERE - server_id = $1 AND - destroy_ts IS NULL - FOR UPDATE - ", - &server_id, - util::timestamp::now(), - ) - .await?; - - let Some(DnsRecordRow { - dns_record_id, - secondary_dns_record_id, - }) = row - else { - // TODO: This might get stuck in a loop if does not finish installing - // NOTE: It is safe to do nothing in this case because both this worker and - // `cluster-server-dns-create` use transactions - retry_bail!("server has no dns records yet"); - }; - - let zone_id = unwrap!(util::env::cloudflare::zone::job::id(), "dns not configured"); - - // Delete main record - if let Some(record_id) = dns_record_id { - let res = client - .request(&cf::dns::DeleteDnsRecord { - zone_identifier: zone_id, - identifier: &record_id, - }) - .await; - - if let Err(cf_framework::response::ApiFailure::Error( - http::status::StatusCode::NOT_FOUND, - _, - )) = res - { - tracing::warn!(%zone_id, %record_id, "dns record not found"); - } else { - res?; - tracing::info!(%record_id, "deleted dns record"); - } - } else { - tracing::warn!("server has no primary dns record"); - } - - // Delete secondary record - if let Some(record_id) = secondary_dns_record_id { - let res = client - .request(&cf::dns::DeleteDnsRecord { - zone_identifier: zone_id, - identifier: &record_id, - }) - .await; - - if let Err(cf_framework::response::ApiFailure::Error( - http::status::StatusCode::NOT_FOUND, - _, - )) = res - { - tracing::warn!(%zone_id, %record_id, "secondary dns record not found"); - } else { - res?; - tracing::info!(%record_id, "deleted secondary dns record"); - } - } else { - tracing::warn!("server has no secondary dns record"); - } - - // Update db record - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.servers_cloudflare - SET destroy_ts = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - util::timestamp::now(), - ) - .await?; - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_drain.rs b/svc/pkg/cluster/worker/src/workers/server_drain.rs deleted file mode 100644 index 121371f66..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_drain.rs +++ /dev/null @@ -1,127 +0,0 @@ -use chirp_worker::prelude::*; -use futures_util::FutureExt; -use nomad_client::{ - apis::{configuration::Configuration, nodes_api}, - models, -}; -use proto::backend::{self, pkg::*}; - -lazy_static::lazy_static! { - static ref NOMAD_CONFIG: Configuration = nomad_util::new_config_from_env().unwrap(); -} - -#[derive(sqlx::FromRow)] -struct Server { - datacenter_id: Uuid, - pool_type: i64, - nomad_node_id: Option, - is_not_draining: bool, -} - -#[worker(name = "cluster-server-drain")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| inner(ctx.clone(), tx).boxed()).await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - let server = sql_fetch_one!( - [ctx, Server, @tx tx] - " - SELECT - datacenter_id, - pool_type, - nomad_node_id, - (drain_ts IS NULL) AS is_not_draining - FROM db_cluster.servers - WHERE server_id = $1 - FOR UPDATE - ", - server_id, - ) - .await?; - - if server.is_not_draining { - tracing::error!("attempting to drain server that was not set as draining"); - return Ok(()); - } - - // Fetch datacenter config - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![server.datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - - let pool_type = unwrap!(backend::cluster::PoolType::from_i32( - server.pool_type as i32 - )); - let pool = unwrap!( - datacenter - .pools - .iter() - .find(|pool| pool.pool_type == server.pool_type as i32), - "missing pool" - ); - - match pool_type { - backend::cluster::PoolType::Job => { - // This worker will never be called if the server has no nomad instance running. This should be an - // unreachable log. - let Some(nomad_node_id) = server.nomad_node_id else { - tracing::error!("server does not have nomad running, cannot drain"); - return Ok(()); - }; - - // Drain complete message is caught by `cluster-nomad-node-drain-complete` - nodes_api::update_node_drain( - &NOMAD_CONFIG, - &nomad_node_id, - models::NodeUpdateDrainRequest { - drain_spec: Some(Box::new(models::DrainSpec { - // In nanoseconds. `drain_timeout` must be below 292 years for this to not overflow - deadline: Some((pool.drain_timeout * 1000000) as i64), - ignore_system_jobs: None, - })), - mark_eligible: None, - meta: None, - node_id: Some(nomad_node_id.clone()), - }, - None, - None, - None, - None, - None, - None, - None, - None, - None, - ) - .await?; - - // Prevent new matchmaker requests to the node running on this server - msg!([ctx] mm::msg::nomad_node_closed_set(&nomad_node_id) { - datacenter_id: Some(server.datacenter_id.into()), - nomad_node_id: nomad_node_id.clone(), - is_closed: true, - }) - .await?; - } - backend::cluster::PoolType::Gg => { - // Delete DNS record - msg!([ctx] cluster::msg::server_dns_delete(server_id) { - server_id: Some(server_id.into()), - }) - .await?; - } - backend::cluster::PoolType::Ats => {} - } - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_install_complete.rs b/svc/pkg/cluster/worker/src/workers/server_install_complete.rs deleted file mode 100644 index 138bb73ce..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_install_complete.rs +++ /dev/null @@ -1,24 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; - -#[worker(name = "cluster-server-install-complete")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let provider = unwrap!(backend::cluster::Provider::from_i32(ctx.provider)); - - // No server id means this was from a prebake install - if ctx.server_id.is_none() { - match provider { - backend::cluster::Provider::Linode => { - msg!([ctx] linode::msg::prebake_install_complete(&ctx.public_ip) { - public_ip: ctx.public_ip.clone(), - datacenter_id: ctx.datacenter_id, - }) - .await?; - } - } - } - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_provision.rs b/svc/pkg/cluster/worker/src/workers/server_provision.rs deleted file mode 100644 index 2376d670d..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_provision.rs +++ /dev/null @@ -1,314 +0,0 @@ -use std::net::{IpAddr, Ipv4Addr}; - -use chirp_worker::prelude::*; -use futures_util::FutureExt; -use proto::backend::{self, cluster::PoolType, pkg::*}; -use rand::Rng; -use util_cluster::metrics; - -struct ProvisionResponse { - provider_server_id: String, - provider_hardware: String, - public_ip: String, - already_installed: bool, -} - -// More than the timeout in linode-server-provision -#[worker(name = "cluster-server-provision", timeout = 300)] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - // TODO: RVTEE-75 - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| inner(ctx.clone(), tx).boxed()).await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, -) -> GlobalResult<()> { - let datacenter_id = unwrap!(ctx.datacenter_id).as_uuid(); - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - let pool_type = unwrap!(backend::cluster::PoolType::from_i32(ctx.pool_type)); - let provider = unwrap!(backend::cluster::Provider::from_i32(ctx.provider)); - - // Check if server is already provisioned - // NOTE: sql record already exists before this worker is called - let (provider_server_id,) = sql_fetch_one!( - [ctx, (Option,)] - " - SELECT - provider_server_id - FROM db_cluster.servers - WHERE server_id = $1 - ", - server_id, - ) - .await?; - if let Some(provider_server_id) = provider_server_id { - tracing::warn!( - ?server_id, - ?provider_server_id, - "server is already provisioned" - ); - return Ok(()); - } - - // Fetch datacenter config - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - let pool = unwrap!( - datacenter - .pools - .iter() - .find(|p| p.pool_type == ctx.pool_type), - "datacenter does not have this type of pool configured" - ); - - // Get a new vlan ip - let vlan_ip = get_vlan_ip(&ctx, tx, datacenter_id, server_id, pool_type).await?; - - sql_execute!( - [ctx] - " - UPDATE db_cluster.servers - SET vlan_ip = $2 - WHERE server_id = $1 - ", - server_id, - IpAddr::V4(vlan_ip), - ) - .await?; - - // Iterate through list of hardware and attempt to schedule a server. Goes to the next - // hardware if an error happens during provisioning - let mut hardware_list = pool.hardware.iter(); - let provision_res = loop { - // List exhausted - let Some(hardware) = hardware_list.next() else { - break None; - }; - - tracing::info!( - "attempting to provision hardware: {}", - hardware.provider_hardware - ); - - match provider { - backend::cluster::Provider::Linode => { - let res = op!([ctx] linode_server_provision { - datacenter_id: ctx.datacenter_id, - server_id: ctx.server_id, - provider_datacenter_id: datacenter.provider_datacenter_id.clone(), - hardware: Some(hardware.clone()), - pool_type: ctx.pool_type, - vlan_ip: vlan_ip.to_string(), - tags: ctx.tags.clone(), - use_prebakes: datacenter.prebakes_enabled, - }) - .await; - - match res { - Ok(res) => { - break Some(ProvisionResponse { - provider_server_id: res.provider_server_id.clone(), - provider_hardware: hardware.provider_hardware.clone(), - public_ip: res.public_ip.clone(), - already_installed: res.already_installed, - }) - } - Err(err) => { - tracing::error!( - ?err, - ?server_id, - "failed to provision server, cleaning up" - ); - - cleanup(&ctx, server_id).await?; - } - } - } - } - }; - - if let Some(provision_res) = provision_res { - let provision_complete_ts = util::timestamp::now(); - - let (create_ts,) = sql_fetch_one!( - [ctx, (i64,)] - " - UPDATE db_cluster.servers - SET - provider_server_id = $2, - provider_hardware = $3, - public_ip = $4, - provision_complete_ts = $5, - install_complete_ts = $6 - WHERE server_id = $1 - RETURNING create_ts - ", - server_id, - &provision_res.provider_server_id, - &provision_res.provider_hardware, - &provision_res.public_ip, - provision_complete_ts, - if provision_res.already_installed { - Some(provision_complete_ts) - } else { - None - }, - ) - .await?; - - if provision_res.already_installed { - // Create DNS record because the server is already installed - if let backend::cluster::PoolType::Gg = pool_type { - // Source of truth record - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.servers_cloudflare (server_id) - VALUES ($1) - ", - server_id, - ) - .await?; - - msg!([ctx] cluster::msg::server_dns_create(server_id) { - server_id: ctx.server_id, - }) - .await?; - } - } - // Install components on server - else { - let request_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::server_install(request_id) { - request_id: Some(request_id.into()), - public_ip: provision_res.public_ip, - datacenter_id: ctx.datacenter_id, - server_id: ctx.server_id, - pool_type: ctx.pool_type, - provider: ctx.provider, - initialize_immediately: true, - }) - .await?; - } - - insert_metrics(datacenter, &pool_type, provision_complete_ts, create_ts).await?; - } else { - tracing::error!(?server_id, hardware_options=?pool.hardware.len(), "failed all attempts to provision server"); - bail!("failed all attempts to provision server"); - } - - Ok(()) -} - -async fn get_vlan_ip( - ctx: &OperationContext, - _tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - datacenter_id: Uuid, - server_id: Uuid, - pool_type: backend::cluster::PoolType, -) -> GlobalResult { - // Find next available vlan index - let mut vlan_addr_range = match pool_type { - PoolType::Job => util::net::job::vlan_addr_range(), - PoolType::Gg => util::net::gg::vlan_addr_range(), - PoolType::Ats => util::net::ats::vlan_addr_range(), - }; - let max_idx = vlan_addr_range.count() as i64; - let (network_idx,) = sql_fetch_one!( - [ctx, (i64,)] - " - WITH - get_next_network_idx AS ( - SELECT mod(idx + $1, $2) AS idx - FROM generate_series(0, $2) AS s(idx) - WHERE NOT EXISTS ( - SELECT 1 - FROM db_cluster.servers - WHERE - pool_type = $3 AND - -- Technically this should check all servers where their datacenter's provider and - -- provider_datacenter_id are the same because VLAN is separated by irl datacenter - -- but this is good enough - datacenter_id = $4 AND - network_idx = mod(idx + $1, $2) AND - cloud_destroy_ts IS NULL - ) - LIMIT 1 - ), - update_network_idx AS ( - UPDATE db_cluster.servers - SET network_idx = (SELECT idx FROM get_next_network_idx) - WHERE server_id = $5 - RETURNING 1 - ) - SELECT idx FROM get_next_network_idx - ", - // Choose a random index to start from for better index spread - rand::thread_rng().gen_range(0i64..max_idx), - max_idx, - pool_type as i64, - datacenter_id, - server_id, - ) - .await?; - - let vlan_ip = unwrap!(vlan_addr_range.nth(network_idx as usize)); - - Ok(vlan_ip) -} - -// This function is used to destroy leftovers from a failed partial provision. -async fn cleanup( - ctx: &OperationContext, - server_id: Uuid, -) -> GlobalResult<()> { - // NOTE: Usually before publishing this message we would set `cloud_destroy_ts`. We do not set it here - // because this message will be retried with the same server id - - // Wait for server to complete destroying so we don't get a primary key conflict (the same server id - // will be used to try and provision the next hardware option) - msg!([ctx] cluster::msg::server_destroy(server_id) -> cluster::msg::server_destroy_complete { - server_id: Some(server_id.into()), - // We force destroy because the provision process failed - force: true, - }) - .await?; - - Ok(()) -} - -async fn insert_metrics( - dc: &backend::cluster::Datacenter, - pool_type: &backend::cluster::PoolType, - provision_complete_ts: i64, - create_ts: i64, -) -> GlobalResult<()> { - let datacenter_id = unwrap_ref!(dc.datacenter_id).as_uuid().to_string(); - let cluster_id = unwrap_ref!(dc.cluster_id).as_uuid().to_string(); - let dt = (provision_complete_ts - create_ts) as f64 / 1000.0; - - metrics::PROVISION_DURATION - .with_label_values(&[ - cluster_id.as_str(), - datacenter_id.as_str(), - &dc.provider_datacenter_id, - &dc.name_id, - match pool_type { - backend::cluster::PoolType::Job => "job", - backend::cluster::PoolType::Gg => "gg", - backend::cluster::PoolType::Ats => "ats", - }, - ]) - .observe(dt); - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_taint.rs b/svc/pkg/cluster/worker/src/workers/server_taint.rs deleted file mode 100644 index e3ec0206a..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_taint.rs +++ /dev/null @@ -1,93 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; -use std::collections::HashSet; - -#[worker(name = "cluster-server-taint")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - let filter = unwrap_ref!(ctx.filter); - - let server_ids = if filter.filter_server_ids { - Some( - filter - .server_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let cluster_ids = if filter.filter_cluster_ids { - Some( - filter - .cluster_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let datacenter_ids = if filter.filter_datacenter_ids { - Some( - filter - .datacenter_ids - .iter() - .map(|&x| x.into()) - .collect::>(), - ) - } else { - None - }; - let pool_types = if filter.filter_pool_types { - Some(&filter.pool_types) - } else { - None - }; - let public_ips = if filter.filter_public_ips { - Some(&filter.public_ips) - } else { - None - }; - - // Taint server records. These will be incrementally drained and destroyed by `cluster-datacenter-scale` - let updated_dc_ids = sql_fetch_all!( - [ctx, (Uuid,)] - " - UPDATE db_cluster.servers AS s - SET taint_ts = $1 - FROM db_cluster.datacenters AS d - WHERE - s.datacenter_id = d.datacenter_id - AND s.taint_ts IS NULL - AND s.cloud_destroy_ts IS NULL - AND ($2 IS NULL OR s.server_id = ANY($2)) - AND ($3 IS NULL OR d.cluster_id = ANY($3)) - AND ($4 IS NULL OR s.datacenter_id = ANY($4)) - AND ($5 IS NULL OR s.pool_type = ANY($5)) - AND ($6 IS NULL OR s.public_ip = ANY($6::inet[])) - RETURNING s.datacenter_id - ", - util::timestamp::now(), - &server_ids, - &cluster_ids, - &datacenter_ids, - &pool_types, - &public_ips, - ) - .await?; - - // Trigger rescale in affected datacenters - let updated_dc_ids = updated_dc_ids - .into_iter() - .map(|x| x.0) - .collect::>(); - for dc_id in updated_dc_ids { - msg!([ctx] cluster::msg::datacenter_scale(dc_id) { - datacenter_id: Some(dc_id.into()), - }) - .await?; - } - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/src/workers/server_undrain.rs b/svc/pkg/cluster/worker/src/workers/server_undrain.rs deleted file mode 100644 index d00adb287..000000000 --- a/svc/pkg/cluster/worker/src/workers/server_undrain.rs +++ /dev/null @@ -1,122 +0,0 @@ -use chirp_worker::prelude::*; -use futures_util::FutureExt; -use nomad_client::{ - apis::{configuration::Configuration, nodes_api}, - models, -}; -use proto::backend::{self, pkg::*}; - -lazy_static::lazy_static! { - static ref NOMAD_CONFIG: Configuration = nomad_util::new_config_from_env().unwrap(); -} - -#[derive(sqlx::FromRow)] -struct Server { - datacenter_id: Uuid, - pool_type: i64, - nomad_node_id: Option, - is_draining: bool, -} - -#[worker(name = "cluster-server-undrain")] -async fn worker(ctx: &OperationContext) -> GlobalResult<()> { - rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| inner(ctx.clone(), tx).boxed()).await?; - - Ok(()) -} - -async fn inner( - ctx: OperationContext, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, -) -> GlobalResult<()> { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - - // NOTE: `drain_ts` will already be set to null before this worker is called - let server = sql_fetch_one!( - [ctx, Server, @tx tx] - " - SELECT - datacenter_id, - pool_type, - nomad_node_id, - (drain_ts IS NOT NULL) AS is_draining - FROM db_cluster.servers - WHERE server_id = $1 - FOR UPDATE - ", - server_id, - ) - .await?; - - if server.is_draining { - tracing::error!("attempting to undrain server that was not set as undraining"); - return Ok(()); - } - - let pool_type = unwrap!(backend::cluster::PoolType::from_i32( - server.pool_type as i32 - )); - match pool_type { - backend::cluster::PoolType::Job => { - // This worker will never be called if the server has no nomad instance running. This should be an - // unreachable log. - let Some(nomad_node_id) = server.nomad_node_id else { - tracing::error!("server does not have nomad running, cannot undrain"); - return Ok(()); - }; - - nodes_api::update_node_drain( - &NOMAD_CONFIG, - &nomad_node_id, - models::NodeUpdateDrainRequest { - drain_spec: None, - mark_eligible: Some(true), - meta: None, - node_id: Some(nomad_node_id.clone()), - }, - None, - None, - None, - None, - None, - None, - None, - None, - None, - ) - .await?; - - // Allow new matchmaker requests to the node running on this server - msg!([ctx] mm::msg::nomad_node_closed_set(&nomad_node_id) { - datacenter_id: Some(server.datacenter_id.into()), - nomad_node_id: nomad_node_id.clone(), - is_closed: false, - }) - .await?; - } - backend::cluster::PoolType::Gg => { - // Source of truth record - sql_execute!( - [ctx, @tx tx] - " - INSERT INTO db_cluster.servers_cloudflare (server_id) - VALUES ($1) - ", - server_id, - ) - .await?; - - // Recreate DNS record - msg!([ctx] cluster::msg::server_dns_create(server_id) { - server_id: Some(server_id.into()), - }) - .await?; - } - _ => { - // Gracefully fail - tracing::error!("cannot undrain this pool type: {:?}", pool_type); - } - } - - Ok(()) -} diff --git a/svc/pkg/cluster/worker/tests/common.rs b/svc/pkg/cluster/worker/tests/common.rs deleted file mode 100644 index 7af2e14cb..000000000 --- a/svc/pkg/cluster/worker/tests/common.rs +++ /dev/null @@ -1,77 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; - -pub struct Setup { - pub server_id: Uuid, - pub datacenter_id: Uuid, - pub cluster_id: Uuid, - pub pool_type: backend::cluster::PoolType, - pub drain_timeout: u64, -} - -pub struct SetupRes { - pub pools: Vec, - pub provider: backend::cluster::Provider, -} - -pub async fn setup(ctx: &TestCtx, opts: Setup) -> SetupRes { - let pools = vec![backend::cluster::Pool { - pool_type: opts.pool_type as i32, - hardware: vec![backend::cluster::Hardware { - provider_hardware: util_cluster::test::LINODE_HARDWARE.to_string(), - }], - desired_count: 0, - min_count: 0, - max_count: 0, - drain_timeout: opts.drain_timeout, - }]; - let provider = backend::cluster::Provider::Linode; - - msg!([ctx] cluster::msg::create(opts.cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(opts.cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) - .await - .unwrap(); - - msg!([ctx] cluster::msg::datacenter_create(opts.datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(opts.datacenter_id.into()), - cluster_id: Some(opts.cluster_id.into()), - name_id: util::faker::ident(), - display_name: util::faker::ident(), - - provider: provider as i32, - provider_datacenter_id: "us-southeast".to_string(), - provider_api_token: None, - - pools: pools.clone(), - - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) - .await - .unwrap(); - - // Write new server to db - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.servers ( - server_id, - datacenter_id, - pool_type, - create_ts - ) - VALUES ($1, $2, $3, $4) - ", - opts.server_id, - opts.datacenter_id, - opts.pool_type as i64, - util::timestamp::now(), - ) - .await - .unwrap(); - - SetupRes { pools, provider } -} diff --git a/svc/pkg/cluster/worker/tests/create.rs b/svc/pkg/cluster/worker/tests/create.rs deleted file mode 100644 index 0ff58e299..000000000 --- a/svc/pkg/cluster/worker/tests/create.rs +++ /dev/null @@ -1,16 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker_test] -async fn create(ctx: TestCtx) { - let cluster_id = Uuid::new_v4(); - let owner_team_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - owner_team_id: Some(owner_team_id.into()), - name_id: util::faker::ident(), - }) - .await - .unwrap(); -} diff --git a/svc/pkg/cluster/worker/tests/datacenter_create.rs b/svc/pkg/cluster/worker/tests/datacenter_create.rs deleted file mode 100644 index a71564ccc..000000000 --- a/svc/pkg/cluster/worker/tests/datacenter_create.rs +++ /dev/null @@ -1,51 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; - -#[worker_test] -async fn datacenter_create(ctx: TestCtx) { - let datacenter_id = Uuid::new_v4(); - let cluster_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) - .await - .unwrap(); - - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - display_name: util::faker::ident(), - - provider: backend::cluster::Provider::Linode as i32, - provider_datacenter_id: "us-southeast".to_string(), - provider_api_token: None, - - pools: Vec::new(), - - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) - .await - .unwrap(); - - // Check if tls record exists - let (exists,) = sql_fetch_one!( - [ctx, (bool,)] - " - SELECT EXISTS ( - SELECT 1 - FROM db_cluster.datacenter_tls - WHERE datacenter_id = $1 - ) - ", - datacenter_id, - ) - .await - .unwrap(); - - assert!(exists, "no tls record"); -} diff --git a/svc/pkg/cluster/worker/tests/game_link.rs b/svc/pkg/cluster/worker/tests/game_link.rs deleted file mode 100644 index 6364b054a..000000000 --- a/svc/pkg/cluster/worker/tests/game_link.rs +++ /dev/null @@ -1,15 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::pkg::*; - -#[worker_test] -async fn create(ctx: TestCtx) { - let game_id = Uuid::new_v4(); - let cluster_id = Uuid::new_v4(); - - msg!([ctx] cluster::msg::game_link(game_id, cluster_id) -> cluster::msg::game_link_complete { - game_id: Some(game_id.into()), - cluster_id: Some(cluster_id.into()), - }) - .await - .unwrap(); -} diff --git a/svc/pkg/game/ops/create/Cargo.toml b/svc/pkg/game/ops/create/Cargo.toml index 16855fbc6..efce2a1cd 100644 --- a/svc/pkg/game/ops/create/Cargo.toml +++ b/svc/pkg/game/ops/create/Cargo.toml @@ -6,11 +6,13 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } prost = "0.10" +rivet-operation = { path = "../../../../../lib/operation/core" } util-team = { package = "rivet-util-team", path = "../../../team/util" } +cluster = { path = "../../../cluster" } game-validate = { path = "../validate" } team-get = { path = "../../../team/ops/get" } diff --git a/svc/pkg/game/ops/create/src/lib.rs b/svc/pkg/game/ops/create/src/lib.rs index 21cee15b5..d2153613d 100644 --- a/svc/pkg/game/ops/create/src/lib.rs +++ b/svc/pkg/game/ops/create/src/lib.rs @@ -66,11 +66,15 @@ async fn handle( .await?; if let Some(cluster_id) = ctx.cluster_id { - msg!([ctx] cluster::msg::game_link(game_id, cluster_id) -> cluster::msg::game_link_complete { - game_id: Some(game_id.into()), - cluster_id: ctx.cluster_id, - }) - .await?; + chirp_workflow::compat::tagged_signal( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id.as_uuid(), + }), + cluster::workflows::cluster::GameLink { game_id }, + ) + .await + .unwrap(); } msg!([ctx] game::msg::create_complete(game_id) { diff --git a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Cargo.toml b/svc/pkg/linode/Cargo.toml similarity index 51% rename from svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Cargo.toml rename to svc/pkg/linode/Cargo.toml index 0ff168c36..d1f1d82e7 100644 --- a/svc/pkg/cluster/ops/datacenter-resolve-for-name-id/Cargo.toml +++ b/svc/pkg/linode/Cargo.toml @@ -1,14 +1,18 @@ [package] -name = "cluster-datacenter-resolve-for-name-id" +name = "linode" version = "0.0.1" edition = "2018" authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -prost = "0.10" -rivet-operation = { path = "../../../../../lib/operation/core" } +chirp-workflow = { path = "../../../lib/chirp-workflow/core" } +chrono = "0.4" +rand = "0.8" +reqwest = { version = "0.11", features = ["json"] } +serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" +ssh-key = "0.6.3" [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" @@ -16,4 +20,4 @@ rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" default-features = false [dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } +cluster = { path = "../cluster" } diff --git a/svc/pkg/linode/ops/server-provision/Service.toml b/svc/pkg/linode/Service.toml similarity index 73% rename from svc/pkg/linode/ops/server-provision/Service.toml rename to svc/pkg/linode/Service.toml index 40485ec54..1cc556583 100644 --- a/svc/pkg/linode/ops/server-provision/Service.toml +++ b/svc/pkg/linode/Service.toml @@ -1,10 +1,10 @@ [service] -name = "linode-server-provision" +name = "linode" [runtime] kind = "rust" -[operation] +[package] [secrets] "linode/token" = { optional = true } diff --git a/svc/pkg/linode/db/linode/Service.toml b/svc/pkg/linode/db/linode/Service.toml new file mode 100644 index 000000000..4c08d1950 --- /dev/null +++ b/svc/pkg/linode/db/linode/Service.toml @@ -0,0 +1,7 @@ +[service] +name = "db-linode" + +[runtime] +kind = "crdb" + +[database] diff --git a/svc/pkg/linode/db/linode/migrations/20240711200008_init.down.sql b/svc/pkg/linode/db/linode/migrations/20240711200008_init.down.sql new file mode 100644 index 000000000..e69de29bb diff --git a/svc/pkg/linode/db/linode/migrations/20240711200008_init.up.sql b/svc/pkg/linode/db/linode/migrations/20240711200008_init.up.sql new file mode 100644 index 000000000..ff17d254c --- /dev/null +++ b/svc/pkg/linode/db/linode/migrations/20240711200008_init.up.sql @@ -0,0 +1,4 @@ +CREATE TABLE server_images ( + image_id STRING PRIMARY KEY, + complete_ts INT +); diff --git a/svc/pkg/linode/ops/instance-type-get/Cargo.toml b/svc/pkg/linode/ops/instance-type-get/Cargo.toml deleted file mode 100644 index fb8fa5f83..000000000 --- a/svc/pkg/linode/ops/instance-type-get/Cargo.toml +++ /dev/null @@ -1,20 +0,0 @@ -[package] -name = "linode-instance-type-get" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -util-linode = { package = "rivet-util-linode", path = "../../util" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } diff --git a/svc/pkg/linode/ops/instance-type-get/Service.toml b/svc/pkg/linode/ops/instance-type-get/Service.toml deleted file mode 100644 index 1d9736733..000000000 --- a/svc/pkg/linode/ops/instance-type-get/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "linode-instance-type-get" - -[runtime] -kind = "rust" - -[operation] - -[secrets] -"linode/token" = { optional = true } diff --git a/svc/pkg/linode/ops/instance-type-get/src/lib.rs b/svc/pkg/linode/ops/instance-type-get/src/lib.rs deleted file mode 100644 index 4a83d386e..000000000 --- a/svc/pkg/linode/ops/instance-type-get/src/lib.rs +++ /dev/null @@ -1,43 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; -use util_linode::api; - -#[operation(name = "linode-instance-type-get")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - // Build HTTP client - let client = util_linode::Client::new(None).await?; - - // Get hardware stats from linode and cache - let instance_types_res = ctx - .cache() - .ttl(util::duration::days(1)) - .fetch_one_proto("instance_types", "linode", { - let client = client.clone(); - move |mut cache, key| { - let client = client.clone(); - async move { - let api_res = api::list_instance_types(&client).await?; - - cache.resolve( - &key, - linode::instance_type_get::CacheInstanceTypes { - instance_types: api_res.into_iter().map(Into::into).collect::>(), - }, - ); - - Ok(cache) - } - } - }) - .await?; - - let instance_types = unwrap!(instance_types_res) - .instance_types - .into_iter() - .filter(|ty| ctx.hardware_ids.iter().any(|h| h == &ty.hardware_id)) - .collect::>(); - - Ok(linode::instance_type_get::Response { instance_types }) -} diff --git a/svc/pkg/linode/ops/server-destroy/Cargo.toml b/svc/pkg/linode/ops/server-destroy/Cargo.toml deleted file mode 100644 index 23c155ed3..000000000 --- a/svc/pkg/linode/ops/server-destroy/Cargo.toml +++ /dev/null @@ -1,25 +0,0 @@ -[package] -name = "linode-server-destroy" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -reqwest = { version = "0.11", features = ["json"] } -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } -util-linode = { package = "rivet-util-linode", path = "../../util" } - -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } - -linode-server-provision = { path = "../server-provision" } diff --git a/svc/pkg/linode/ops/server-destroy/Service.toml b/svc/pkg/linode/ops/server-destroy/Service.toml deleted file mode 100644 index be0e245fc..000000000 --- a/svc/pkg/linode/ops/server-destroy/Service.toml +++ /dev/null @@ -1,10 +0,0 @@ -[service] -name = "linode-server-destroy" - -[runtime] -kind = "rust" - -[operation] - -[secrets] -"linode/token" = { optional = true } diff --git a/svc/pkg/linode/ops/server-destroy/src/lib.rs b/svc/pkg/linode/ops/server-destroy/src/lib.rs deleted file mode 100644 index e90978052..000000000 --- a/svc/pkg/linode/ops/server-destroy/src/lib.rs +++ /dev/null @@ -1,72 +0,0 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; -use util_linode::api; - -#[derive(sqlx::FromRow)] -struct LinodeData { - ssh_key_id: i64, - linode_id: Option, - firewall_id: Option, -} - -#[operation(name = "linode-server-destroy")] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - let datacenter_id = unwrap!(ctx.datacenter_id); - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - - let data = sql_fetch_optional!( - [ctx, LinodeData] - " - SELECT ssh_key_id, linode_id, firewall_id - FROM db_cluster.servers_linode - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - ) - .await?; - - let Some(data) = data else { - tracing::warn!("deleting server that doesn't exist"); - return Ok(linode::server_destroy::Response {}); - }; - - // Build HTTP client - let client = util_linode::Client::new(datacenter.provider_api_token.clone()).await?; - - if let Some(linode_id) = data.linode_id { - api::delete_instance(&client, linode_id).await?; - } - - api::delete_ssh_key(&client, data.ssh_key_id).await?; - - if let Some(firewall_id) = data.firewall_id { - api::delete_firewall(&client, firewall_id).await?; - } - - // Remove record - sql_execute!( - [ctx] - " - UPDATE db_cluster.servers_linode - SET destroy_ts = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - util::timestamp::now(), - ) - .await?; - - Ok(linode::server_destroy::Response {}) -} diff --git a/svc/pkg/linode/ops/server-provision/Cargo.toml b/svc/pkg/linode/ops/server-provision/Cargo.toml deleted file mode 100644 index a7e25cafb..000000000 --- a/svc/pkg/linode/ops/server-provision/Cargo.toml +++ /dev/null @@ -1,25 +0,0 @@ -[package] -name = "linode-server-provision" -version = "0.0.1" -edition = "2018" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chirp-client = { path = "../../../../../lib/chirp/client" } -rivet-operation = { path = "../../../../../lib/operation/core" } -reqwest = { version = "0.11", features = ["json"] } -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } -util-linode = { package = "rivet-util-linode", path = "../../util" } - -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } - -[dependencies.sqlx] -git = "https://github.com/rivet-gg/sqlx" -rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" -default-features = false - -[dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } - -linode-server-destroy = { path = "../server-destroy" } diff --git a/svc/pkg/linode/ops/server-provision/README.md b/svc/pkg/linode/ops/server-provision/README.md deleted file mode 100644 index 7b0f7f2d1..000000000 --- a/svc/pkg/linode/ops/server-provision/README.md +++ /dev/null @@ -1,5 +0,0 @@ -# linode-server-provision - -This was meant to be agnostic to all other packages and simply create a server on Linode, but because of -custom API keys and prebake images we need to include a `datacenter_id` in the request. In the future and if -needed this can be made optional so that this endpoint does not require a `datacenter_id`. diff --git a/svc/pkg/linode/ops/server-provision/src/lib.rs b/svc/pkg/linode/ops/server-provision/src/lib.rs deleted file mode 100644 index d08dd1c2f..000000000 --- a/svc/pkg/linode/ops/server-provision/src/lib.rs +++ /dev/null @@ -1,266 +0,0 @@ -use proto::backend::{self, cluster::PoolType, pkg::*}; -use rivet_operation::prelude::*; -use util_linode::api; - -// Less than the timeout in cluster-server-provision -#[operation(name = "linode-server-provision", timeout = 245)] -pub async fn handle( - ctx: OperationContext, -) -> GlobalResult { - let crdb = ctx.crdb().await?; - let server_id = unwrap_ref!(ctx.server_id).as_uuid(); - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - let provider_datacenter_id = ctx.provider_datacenter_id.clone(); - let pool_type = unwrap!(PoolType::from_i32(ctx.pool_type)); - let provider_hardware = unwrap_ref!(ctx.hardware).provider_hardware.clone(); - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - - let ns = util::env::namespace(); - let pool_type_str = match pool_type { - PoolType::Job => "job", - PoolType::Gg => "gg", - PoolType::Ats => "ats", - }; - // Linode label must be 3-64 characters, UUID's are 36 - let name = format!("{ns}-{server_id}"); - - let tags = ctx - .tags - .iter() - .cloned() - .chain([ - // HACK: Linode requires tags to be > 3 characters. We extend the namespace to make sure it - // meets the minimum length requirement. - format!("rivet-{ns}"), - format!("{ns}-{provider_datacenter_id}"), - format!("{ns}-{pool_type_str}"), - format!("{ns}-{provider_datacenter_id}-{pool_type_str}"), - ]) - .collect::>(); - - let firewall_inbound = match pool_type { - PoolType::Job => util::net::job::firewall(), - PoolType::Gg => util::net::gg::firewall(), - PoolType::Ats => util::net::ats::firewall(), - }; - - // Build context - let server = api::ProvisionCtx { - datacenter: provider_datacenter_id, - name, - hardware: provider_hardware, - vlan_ip: Some(ctx.vlan_ip.clone()), - tags, - firewall_inbound, - }; - - // Build HTTP client - let client = util_linode::Client::new(datacenter.provider_api_token.clone()).await?; - - // Create SSH key - let ssh_key_label = format!("{ns}-{server_id}"); - let ssh_key_res = api::create_ssh_key( - &client, - &ssh_key_label, - ctx.tags.iter().any(|tag| tag == "test"), - ) - .await?; - - // Write SSH key id - sql_execute!( - [ctx, &crdb] - " - INSERT INTO db_cluster.servers_linode ( - server_id, - ssh_key_id - ) - VALUES ($1, $2) - ", - server_id, - ssh_key_res.id as i64, - ) - .await?; - - let create_instance_res = - api::create_instance(&client, &server, &ssh_key_res.public_key).await?; - let linode_id = create_instance_res.id; - - // Write linode id - sql_execute!( - [ctx, &crdb] - " - UPDATE db_cluster.servers_linode - SET linode_id = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - linode_id as i64, - ) - .await?; - - api::wait_instance_ready(&client, linode_id).await?; - - let (create_disks_res, used_custom_image) = create_disks( - &ctx, - &crdb, - &client, - CreateDisks { - provider_datacenter_id: &server.datacenter, - datacenter_id, - pool_type, - ssh_key: &ssh_key_res.public_key, - linode_id, - server_disk_size: create_instance_res.specs.disk, - }, - ) - .await?; - - api::create_instance_config(&client, &server, linode_id, &create_disks_res).await?; - - let firewall_res = api::create_firewall(&client, &server, linode_id).await?; - - // Write firewall id - sql_execute!( - [ctx, &crdb] - " - UPDATE db_cluster.servers_linode - SET firewall_id = $2 - WHERE - server_id = $1 AND - destroy_ts IS NULL - ", - server_id, - firewall_res.id as i64, - ) - .await?; - - api::boot_instance(&client, linode_id).await?; - - let public_ip = api::get_public_ip(&client, linode_id).await?; - - Ok(linode::server_provision::Response { - provider_server_id: linode_id.to_string(), - public_ip: public_ip.to_string(), - already_installed: used_custom_image, - }) -} - -struct CreateDisks<'a> { - provider_datacenter_id: &'a str, - datacenter_id: Uuid, - pool_type: PoolType, - ssh_key: &'a str, - linode_id: u64, - server_disk_size: u64, -} - -async fn create_disks( - ctx: &OperationContext, - crdb: &CrdbPool, - client: &util_linode::Client, - opts: CreateDisks<'_>, -) -> GlobalResult<(api::CreateDisksResponse, bool)> { - // Try to get custom image (if exists) - let (custom_image, updated) = if ctx.use_prebakes { - get_custom_image(ctx, crdb, opts.datacenter_id, opts.pool_type).await? - } else { - (None, false) - }; - - // Default image - let used_custom_image = custom_image.is_some(); - let image = if let Some(custom_image) = custom_image { - tracing::info!("using custom image {}", custom_image); - - custom_image - } else { - tracing::info!("custom image not ready yet, continuing normally"); - - "linode/debian11".to_string() - }; - - // Start custom image creation process - if updated { - msg!([ctx] linode::msg::prebake_provision(opts.datacenter_id, opts.pool_type as i32) { - datacenter_id: ctx.datacenter_id, - pool_type: opts.pool_type as i32, - provider_datacenter_id: opts.provider_datacenter_id.to_string(), - tags: Vec::new(), - }) - .await?; - } - - let create_disks_res = api::create_disks( - client, - opts.ssh_key, - opts.linode_id, - &image, - opts.server_disk_size, - ) - .await?; - - Ok((create_disks_res, used_custom_image)) -} - -async fn get_custom_image( - ctx: &OperationContext, - crdb: &CrdbPool, - datacenter_id: Uuid, - pool_type: PoolType, -) -> GlobalResult<(Option, bool)> { - let provider = backend::cluster::Provider::Linode; - - // Get the custom image id for this server, or insert a record and start creating one - let (image_id, updated) = sql_fetch_one!( - [ctx, (Option, bool), &crdb] - " - WITH - updated AS ( - INSERT INTO db_cluster.server_images AS s ( - provider, install_hash, datacenter_id, pool_type, create_ts - ) - VALUES ($1, $2, $3, $4, $5) - ON CONFLICT (provider, install_hash, datacenter_id, pool_type) DO UPDATE - SET - provider_image_id = NULL, - create_ts = $5 - WHERE s.create_ts < $6 - RETURNING provider, install_hash, datacenter_id, pool_type - ), - selected AS ( - SELECT provider, install_hash, datacenter_id, pool_type, provider_image_id - FROM db_cluster.server_images - WHERE - provider = $1 AND - install_hash = $2 AND - datacenter_id = $3 AND - pool_type = $4 - ) - SELECT - selected.provider_image_id, - -- Primary key is not null - (updated.provider IS NOT NULL) AS updated - FROM selected - FULL OUTER JOIN updated - ON true - ", - provider as i64, - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - pool_type as i64, - util::timestamp::now(), - // 5 month expiration - util::timestamp::now() - util::duration::days(5 * 30), - ) - .await?; - - // Updated is true if this specific sql call either reset (if expired) or inserted the row - Ok((if updated { None } else { image_id }, updated)) -} diff --git a/svc/pkg/linode/proto/instance-type-get.proto b/svc/pkg/linode/proto/instance-type-get.proto deleted file mode 100644 index 81d569450..000000000 --- a/svc/pkg/linode/proto/instance-type-get.proto +++ /dev/null @@ -1,26 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.linode.instance_type_get; - -import "proto/common.proto"; - -message Request { - repeated string hardware_ids = 1; -} - -message Response { - message InstanceType { - string hardware_id = 1; - uint64 memory = 2; - uint64 disk = 3; - uint64 vcpus = 4; - uint64 transfer = 5; - } - - repeated InstanceType instance_types = 1; -} - -message CacheInstanceTypes { - repeated Response.InstanceType instance_types = 1; -} - diff --git a/svc/pkg/linode/proto/msg/prebake-install-complete.proto b/svc/pkg/linode/proto/msg/prebake-install-complete.proto deleted file mode 100644 index 45d681e4e..000000000 --- a/svc/pkg/linode/proto/msg/prebake-install-complete.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.linode.msg.prebake_install_complete; - -import "proto/common.proto"; - -/// name = "msg-linode-prebake-install-complete" -/// parameters = [ -/// { name = "public_ip" }, -/// ] -message Message { - string public_ip = 1; - rivet.common.Uuid datacenter_id = 2; -} diff --git a/svc/pkg/linode/proto/msg/prebake-provision.proto b/svc/pkg/linode/proto/msg/prebake-provision.proto deleted file mode 100644 index e6045bb57..000000000 --- a/svc/pkg/linode/proto/msg/prebake-provision.proto +++ /dev/null @@ -1,18 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.linode.msg.prebake_provision; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -/// name = "msg-linode-prebake-provision" -/// parameters = [ -/// { name = "datacenter_id" }, -/// { name = "pool_type" }, -/// ] -message Message { - rivet.common.Uuid datacenter_id = 1; - rivet.backend.cluster.PoolType pool_type = 2; - string provider_datacenter_id = 3; - repeated string tags = 4; -} diff --git a/svc/pkg/linode/proto/server-destroy.proto b/svc/pkg/linode/proto/server-destroy.proto deleted file mode 100644 index 430e302a9..000000000 --- a/svc/pkg/linode/proto/server-destroy.proto +++ /dev/null @@ -1,14 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.linode.server_destroy; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - rivet.common.Uuid server_id = 1; - rivet.common.Uuid datacenter_id = 2; -} - -message Response { -} diff --git a/svc/pkg/linode/proto/server-provision.proto b/svc/pkg/linode/proto/server-provision.proto deleted file mode 100644 index 81eabb0d2..000000000 --- a/svc/pkg/linode/proto/server-provision.proto +++ /dev/null @@ -1,23 +0,0 @@ -syntax = "proto3"; - -package rivet.backend.pkg.linode.server_provision; - -import "proto/common.proto"; -import "proto/backend/cluster.proto"; - -message Request { - rivet.common.Uuid datacenter_id = 7; - rivet.common.Uuid server_id = 1; - string provider_datacenter_id = 2; - rivet.backend.cluster.Hardware hardware = 3; - rivet.backend.cluster.PoolType pool_type = 4; - string vlan_ip = 5; - repeated string tags = 6; - bool use_prebakes = 8; -} - -message Response { - string provider_server_id = 1; - string public_ip = 2; - bool already_installed = 3; -} diff --git a/svc/pkg/linode/src/lib.rs b/svc/pkg/linode/src/lib.rs new file mode 100644 index 000000000..1ebf758e3 --- /dev/null +++ b/svc/pkg/linode/src/lib.rs @@ -0,0 +1,15 @@ +use chirp_workflow::prelude::*; + +pub mod ops; +pub mod types; +pub mod util; +pub mod workflows; + +pub fn registry() -> Registry { + use workflows::*; + + let mut registry = Registry::new(); + registry.register_workflow::(); + + registry +} diff --git a/svc/pkg/linode/src/ops/instance_type_get.rs b/svc/pkg/linode/src/ops/instance_type_get.rs new file mode 100644 index 000000000..23a72ec5b --- /dev/null +++ b/svc/pkg/linode/src/ops/instance_type_get.rs @@ -0,0 +1,55 @@ +use chirp_workflow::prelude::*; + +use crate::{ + types::InstanceType, + util::{api, client}, +}; + +#[derive(Debug)] +pub struct Input { + pub hardware_ids: Vec, +} + +#[derive(Debug)] +pub struct Output { + pub instance_types: Vec, +} + +#[operation] +pub async fn linode_instance_type_get(ctx: &OperationCtx, input: &Input) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(None).await?; + + // Get hardware stats from linode and cache + let instance_types_res = ctx + .cache() + .ttl(util::duration::days(1)) + .fetch_one_json("instance_types2", "linode", { + let client = client.clone(); + move |mut cache, key| { + let client = client.clone(); + async move { + let api_res = api::list_instance_types(&client).await?; + + cache.resolve( + &key, + api_res + .into_iter() + .map(Into::::into) + .collect::>(), + ); + + Ok(cache) + } + } + }) + .await?; + + // Filter by hardware + let instance_types = unwrap!(instance_types_res) + .into_iter() + .filter(|ty| input.hardware_ids.iter().any(|h| h == &ty.hardware_id)) + .collect::>(); + + Ok(Output { instance_types }) +} diff --git a/svc/pkg/linode/src/ops/mod.rs b/svc/pkg/linode/src/ops/mod.rs new file mode 100644 index 000000000..ff2e5a372 --- /dev/null +++ b/svc/pkg/linode/src/ops/mod.rs @@ -0,0 +1 @@ +pub mod instance_type_get; diff --git a/svc/pkg/linode/src/types.rs b/svc/pkg/linode/src/types.rs new file mode 100644 index 000000000..12f57d922 --- /dev/null +++ b/svc/pkg/linode/src/types.rs @@ -0,0 +1,38 @@ +use chirp_workflow::prelude::*; +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Serialize, Deserialize)] +pub struct InstanceType { + pub hardware_id: String, + pub memory: u64, + pub disk: u64, + pub vcpus: u64, + pub transfer: u64, +} + +#[derive(Debug, Clone, Serialize, Deserialize, Hash)] +pub enum FirewallPreset { + Job, + Gg, + Ats, +} + +impl FirewallPreset { + pub fn rules(&self) -> Vec { + match self { + FirewallPreset::Job => util::net::job::firewall(), + FirewallPreset::Gg => util::net::gg::firewall(), + FirewallPreset::Ats => util::net::ats::firewall(), + } + } +} + +impl std::fmt::Display for FirewallPreset { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + FirewallPreset::Job => write!(f, "job"), + FirewallPreset::Gg => write!(f, "gg"), + FirewallPreset::Ats => write!(f, "ats"), + } + } +} diff --git a/svc/pkg/linode/util/src/api.rs b/svc/pkg/linode/src/util/api.rs similarity index 89% rename from svc/pkg/linode/util/src/api.rs rename to svc/pkg/linode/src/util/api.rs index 6f7f59754..542264b26 100644 --- a/svc/pkg/linode/util/src/api.rs +++ b/svc/pkg/linode/src/util/api.rs @@ -1,22 +1,18 @@ use std::{net::Ipv4Addr, str, time::Duration}; +use chirp_workflow::prelude::*; use chrono::{DateTime, Utc}; -use proto::backend::pkg::*; -use rivet_operation::prelude::*; use serde::{Deserialize, Deserializer}; use serde_json::json; use ssh_key::PrivateKey; -use crate::{generate_password, ApiErrorResponse, Client}; - -pub struct ProvisionCtx { - pub datacenter: String, - pub name: String, - pub hardware: String, - pub vlan_ip: Option, - pub tags: Vec, - pub firewall_inbound: Vec, -} +use crate::{ + types::FirewallPreset, + util::{ + client::{ApiErrorResponse, Client}, + generate_password, + }, +}; #[derive(Deserialize)] struct CreateSshKeyResponse { @@ -79,7 +75,10 @@ pub struct InstanceSpec { pub async fn create_instance( client: &Client, - server: &ProvisionCtx, + name: &str, + datacenter: &str, + hardware: &str, + tags: &[String], ssh_key: &str, ) -> GlobalResult { let ns = util::env::namespace(); @@ -90,12 +89,12 @@ pub async fn create_instance( .post( "/linode/instances", json!({ - "label": server.name, + "label": name, "group": ns, - "region": server.datacenter, - "type": server.hardware, + "region": datacenter, + "type": hardware, "authorized_keys": vec![ssh_key], - "tags": server.tags, + "tags": tags, "private_ip": true, "backups_enabled": false, }), @@ -158,15 +157,16 @@ pub async fn create_disks( pub async fn create_instance_config( client: &Client, - server: &ProvisionCtx, + vlan_ip: Option<&Ipv4Addr>, linode_id: u64, - disks: &CreateDisksResponse, + boot_disk_id: u64, + swap_disk_id: u64, ) -> GlobalResult<()> { tracing::info!("creating instance config"); let ns = util::env::namespace(); - let interfaces = if let Some(vlan_ip) = &server.vlan_ip { + let interfaces = if let Some(vlan_ip) = vlan_ip { let region_vlan = util::net::region::vlan_ip_net(); let ipam_address = format!("{}/{}", vlan_ip, region_vlan.prefix_len()); @@ -196,10 +196,10 @@ pub async fn create_instance_config( "root_device": "/dev/sda", "devices": { "sda": { - "disk_id": disks.boot_id, + "disk_id": boot_disk_id, }, "sdb": { - "disk_id": disks.swap_id, + "disk_id": swap_disk_id }, }, "interfaces": interfaces, @@ -215,15 +215,16 @@ pub struct CreateFirewallResponse { pub async fn create_firewall( client: &Client, - server: &ProvisionCtx, + firewall: &FirewallPreset, + tags: &[String], linode_id: u64, ) -> GlobalResult { tracing::info!("creating firewall"); let ns = util::env::namespace(); - let firewall_inbound = server - .firewall_inbound + let firewall_inbound = firewall + .rules() .iter() .map(|rule| { json!({ @@ -254,7 +255,7 @@ pub async fn create_firewall( "devices": { "linodes": [linode_id], }, - "tags": server.tags, + "tags": tags, }), ) .await @@ -364,7 +365,7 @@ pub async fn get_public_ip(client: &Client, linode_id: u64) -> GlobalResult GlobalResult<()> { +pub async fn delete_ssh_key(client: &Client, ssh_key_id: u64) -> GlobalResult<()> { tracing::info!("deleting linode ssh key"); client @@ -372,7 +373,7 @@ pub async fn delete_ssh_key(client: &Client, ssh_key_id: i64) -> GlobalResult<() .await } -pub async fn delete_instance(client: &Client, linode_id: i64) -> GlobalResult<()> { +pub async fn delete_instance(client: &Client, linode_id: u64) -> GlobalResult<()> { tracing::info!(?linode_id, "deleting linode instance"); client @@ -380,7 +381,7 @@ pub async fn delete_instance(client: &Client, linode_id: i64) -> GlobalResult<() .await } -pub async fn delete_firewall(client: &Client, firewall_id: i64) -> GlobalResult<()> { +pub async fn delete_firewall(client: &Client, firewall_id: u64) -> GlobalResult<()> { tracing::info!("deleting firewall"); client @@ -388,7 +389,7 @@ pub async fn delete_firewall(client: &Client, firewall_id: i64) -> GlobalResult< .await } -pub async fn shut_down(client: &Client, linode_id: i64) -> GlobalResult<()> { +pub async fn shut_down(client: &Client, linode_id: u64) -> GlobalResult<()> { tracing::info!("shutting down instance"); client @@ -407,7 +408,7 @@ pub struct CreateCustomImageResponse { pub async fn create_custom_image( client: &Client, variant: &str, - disk_id: i64, + disk_id: u64, ) -> GlobalResult { tracing::info!("creating custom image"); @@ -446,10 +447,17 @@ pub struct CustomImage { pub async fn list_custom_images(client: &Client) -> GlobalResult> { tracing::info!("listing custom images"); + let ns = util::env::namespace(); + let req = client .inner() .get("https://api.linode.com/v4/images") - .query(&[("page_size", CUSTOM_IMAGE_LIST_SIZE)]); + .query(&[("page_size", CUSTOM_IMAGE_LIST_SIZE)]) + // Filter this namespace only + .header( + "X-Filter", + format!(r#"{{ "label": {{ "+contains": "{ns}-" }} }}"#), + ); let res = client .request(req, None, false) @@ -485,9 +493,9 @@ pub struct InstanceType { pub network_out: u64, } -impl From for linode::instance_type_get::response::InstanceType { +impl From for crate::types::InstanceType { fn from(value: InstanceType) -> Self { - linode::instance_type_get::response::InstanceType { + crate::types::InstanceType { hardware_id: value.id, memory: value.memory, disk: value.disk, diff --git a/svc/pkg/linode/util/src/lib.rs b/svc/pkg/linode/src/util/client.rs similarity index 93% rename from svc/pkg/linode/util/src/lib.rs rename to svc/pkg/linode/src/util/client.rs index 9ee833285..51900cf5e 100644 --- a/svc/pkg/linode/util/src/lib.rs +++ b/svc/pkg/linode/src/util/client.rs @@ -1,13 +1,9 @@ use std::{fmt, time::Duration}; -use rand::{distributions::Alphanumeric, Rng}; +use chirp_workflow::prelude::*; use reqwest::header; -use rivet_operation::prelude::*; use serde::{de::DeserializeOwned, Deserialize}; -pub mod api; -pub mod consts; - #[derive(Clone)] pub struct Client { // Safe to clone, has inner Arc @@ -201,12 +197,3 @@ struct ApiError { field: Option, reason: String, } - -/// Generates a random string for a secret. -pub(crate) fn generate_password(length: usize) -> String { - rand::thread_rng() - .sample_iter(&Alphanumeric) - .take(length) - .map(char::from) - .collect() -} diff --git a/svc/pkg/linode/util/src/consts.rs b/svc/pkg/linode/src/util/consts.rs similarity index 100% rename from svc/pkg/linode/util/src/consts.rs rename to svc/pkg/linode/src/util/consts.rs diff --git a/svc/pkg/cluster/util/src/lib.rs b/svc/pkg/linode/src/util/mod.rs similarity index 53% rename from svc/pkg/cluster/util/src/lib.rs rename to svc/pkg/linode/src/util/mod.rs index 16817b893..4f44d66e7 100644 --- a/svc/pkg/cluster/util/src/lib.rs +++ b/svc/pkg/linode/src/util/mod.rs @@ -1,12 +1,8 @@ -use types::rivet::backend::{self, pkg::*}; -use uuid::Uuid; +use rand::{distributions::Alphanumeric, Rng}; -pub mod metrics; -pub mod test; - -// Use the hash of the server install script in the image variant so that if the install scripts are updated -// we won't be using the old image anymore -pub const INSTALL_SCRIPT_HASH: &str = include_str!(concat!(env!("OUT_DIR"), "/hash.txt")); +pub mod api; +pub mod client; +pub mod consts; // NOTE: We don't reserve CPU because Nomad is running as a higher priority process than the rest and // shouldn't be doing much heavy lifting. @@ -17,9 +13,6 @@ const RESERVE_MEMORY: u64 = RESERVE_SYSTEM_MEMORY + RESERVE_LB_MEMORY; const CPU_PER_CORE: u64 = 1999; -// TTL of the token written to prebake images. Prebake images are renewed before the token would expire -pub const SERVER_TOKEN_TTL: i64 = rivet_util::duration::days(30 * 6); - /// Provider agnostic hardware specs. #[derive(Debug)] pub struct JobNodeConfig { @@ -31,9 +24,7 @@ pub struct JobNodeConfig { } impl JobNodeConfig { - pub fn from_linode( - instance_type: &linode::instance_type_get::response::InstanceType, - ) -> JobNodeConfig { + pub fn from_linode(instance_type: &crate::types::InstanceType) -> JobNodeConfig { // Account for kernel memory overhead // https://www.linode.com/community/questions/17791/why-doesnt-free-m-match-the-full-amount-of-ram-of-my-nanode-plan let memory = instance_type.memory * 96 / 100; @@ -66,22 +57,11 @@ impl JobNodeConfig { } } -// Cluster id for provisioning servers -pub fn default_cluster_id() -> Uuid { - Uuid::nil() -} - -pub fn server_name( - provider_datacenter_id: &str, - pool_type: backend::cluster::PoolType, - server_id: Uuid, -) -> String { - let ns = rivet_util::env::namespace(); - let pool_type_str = match pool_type { - backend::cluster::PoolType::Job => "job", - backend::cluster::PoolType::Gg => "gg", - backend::cluster::PoolType::Ats => "ats", - }; - - format!("{ns}-{provider_datacenter_id}-{pool_type_str}-{server_id}",) +/// Generates a random string for a secret. +pub(crate) fn generate_password(length: usize) -> String { + rand::thread_rng() + .sample_iter(&Alphanumeric) + .take(length) + .map(char::from) + .collect() } diff --git a/svc/pkg/linode/src/workflows/image.rs b/svc/pkg/linode/src/workflows/image.rs new file mode 100644 index 000000000..eda068832 --- /dev/null +++ b/svc/pkg/linode/src/workflows/image.rs @@ -0,0 +1,139 @@ +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::util::{api, client}; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub prebake_server_id: Uuid, + pub api_token: Option, + pub linode_id: u64, + pub boot_disk_id: u64, +} + +#[workflow] +pub async fn linode_image(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + // Shut down server before creating custom image + ctx.activity(ShutdownInput { + linode_id: input.linode_id, + api_token: input.api_token.clone(), + }) + .await?; + + // NOTE: Linode imposes a restriction of 50 characters on custom image labels, so unfortunately we cannot + // use the image variant as the name. All we need from the label is for it to be unique. Keep in mind that + // the UUID and hyphen take 37 characters, leaving us with 13 for the namespace name + let name = format!("{}-{}", util::env::namespace(), input.prebake_server_id); + + let image_id = ctx + .activity(CreateCustomImageInput { + linode_id: input.linode_id, + api_token: input.api_token.clone(), + boot_disk_id: input.boot_disk_id, + name, + }) + .await?; + + // Wait for image to complete creation + let sig = ctx.listen::().await?; + + // Piggy back signal up to the cluster prebake workflow + ctx.tagged_signal( + &json!({ + "server_id": input.prebake_server_id, + }), + sig, + ) + .await?; + + ctx.listen::().await?; + + ctx.activity(DestroyInput { + api_token: input.api_token.clone(), + image_id, + }) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct ShutdownInput { + linode_id: u64, + api_token: Option, +} + +#[activity(Shutdown)] +async fn shutdown(ctx: &ActivityCtx, input: &ShutdownInput) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::shut_down(&client, input.linode_id).await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateCustomImageInput { + linode_id: u64, + api_token: Option, + boot_disk_id: u64, + name: String, +} + +#[activity(CreateCustomImage)] +async fn create_custom_image( + ctx: &ActivityCtx, + input: &CreateCustomImageInput, +) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + let create_image_res = + api::create_custom_image(&client, &input.name, input.boot_disk_id).await?; + + sql_execute!( + [ctx] + " + INSERT INTO db_linode.server_images ( + image_id + ) + VALUES ($1) + ", + &create_image_res.id, + ) + .await?; + + // Add image id to workflow tags + ctx.update_workflow_tags(&json!({ + "linode_id": input.linode_id, + "image_id": create_image_res.id, + })) + .await?; + + Ok(create_image_res.id) +} + +#[signal("linode-image-create-complete")] +pub struct CreateComplete { + pub image_id: String, +} + +#[signal("linode-image-destroy")] +pub struct Destroy {} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct DestroyInput { + api_token: Option, + image_id: String, +} + +#[activity(DestroyActivity)] +async fn destroy(ctx: &ActivityCtx, input: &DestroyInput) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::delete_custom_image(&client, &input.image_id).await?; + + Ok(()) +} diff --git a/svc/pkg/linode/src/workflows/mod.rs b/svc/pkg/linode/src/workflows/mod.rs new file mode 100644 index 000000000..457aecb75 --- /dev/null +++ b/svc/pkg/linode/src/workflows/mod.rs @@ -0,0 +1,2 @@ +pub mod image; +pub mod server; diff --git a/svc/pkg/linode/src/workflows/server.rs b/svc/pkg/linode/src/workflows/server.rs new file mode 100644 index 000000000..fe7e112fa --- /dev/null +++ b/svc/pkg/linode/src/workflows/server.rs @@ -0,0 +1,388 @@ +use std::net::Ipv4Addr; + +use chirp_workflow::prelude::*; +use serde_json::json; + +use crate::{ + types::FirewallPreset, + util::{api, client}, +}; + +const DEFAULT_IMAGE: &str = "linode/debian11"; + +#[derive(Debug, Serialize, Deserialize)] +pub struct Input { + pub server_id: Uuid, + pub provider_datacenter_id: String, + pub custom_image: Option, + pub hardware: String, + pub api_token: Option, + pub firewall_preset: FirewallPreset, + pub vlan_ip: Option, + pub tags: Vec, +} + +#[workflow] +pub async fn linode_server(ctx: &mut WorkflowCtx, input: &Input) -> GlobalResult<()> { + let is_test = input.tags.iter().any(|tag| tag == "test"); + let ns = util::env::namespace(); + // Linode label must be 3-64 characters, UUID's are 36 + let name = format!("{ns}-{}", input.server_id); + + // TODO: Clean up after failure to create + + let tags = input + .tags + .iter() + .cloned() + .chain([ + // HACK: Linode requires tags to be > 3 characters. We extend the namespace to make sure it + // meets the minimum length requirement. + format!("rivet-{ns}"), + format!("{ns}-{}", input.provider_datacenter_id), + format!("{ns}-{}", input.firewall_preset), + format!( + "{ns}-{}-{}", + input.provider_datacenter_id, input.firewall_preset + ), + ]) + .collect::>(); + + let ssh_key_res = ctx + .activity(CreateSshKeyInput { + server_id: input.server_id, + api_token: input.api_token.clone(), + is_test, + }) + .await?; + + let create_instance_res = ctx + .activity(CreateInstanceInput { + api_token: input.api_token.clone(), + ssh_public_key: ssh_key_res.public_key.clone(), + name, + datacenter: input.provider_datacenter_id.clone(), + hardware: input.hardware.clone(), + tags: tags.clone(), + }) + .await?; + + ctx.activity(WaitInstanceReadyInput { + api_token: input.api_token.clone(), + linode_id: create_instance_res.linode_id, + }) + .await?; + + let disks_res = ctx + .activity(CreateDisksInput { + api_token: input.api_token.clone(), + image: input + .custom_image + .clone() + .unwrap_or_else(|| DEFAULT_IMAGE.to_string()), + ssh_public_key: ssh_key_res.public_key.clone(), + linode_id: create_instance_res.linode_id, + disk_size: create_instance_res.server_disk_size, + }) + .await?; + let boot_disk_id = disks_res.boot_id; + + ctx.activity(CreateInstanceConfigInput { + api_token: input.api_token.clone(), + vlan_ip: input.vlan_ip, + linode_id: create_instance_res.linode_id, + disks: disks_res, + }) + .await?; + + let firewall_id = ctx + .activity(CreateFirewallInput { + server_id: input.server_id, + api_token: input.api_token.clone(), + firewall_preset: input.firewall_preset.clone(), + tags, + linode_id: create_instance_res.linode_id, + }) + .await?; + + ctx.activity(BootInstanceInput { + api_token: input.api_token.clone(), + linode_id: create_instance_res.linode_id, + }) + .await?; + + let public_ip = ctx + .activity(GetPublicIpInput { + api_token: input.api_token.clone(), + linode_id: create_instance_res.linode_id, + }) + .await?; + + ctx.tagged_signal( + &json!({ + "server_id": input.server_id, + }), + ProvisionComplete { + linode_id: create_instance_res.linode_id, + public_ip, + boot_disk_id, + }, + ) + .await?; + + ctx.listen::().await?; + + ctx.activity(DestroyInstanceInput { + api_token: input.api_token.clone(), + ssh_key_id: ssh_key_res.ssh_key_id, + linode_id: create_instance_res.linode_id, + firewall_id, + }) + .await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateSshKeyInput { + server_id: Uuid, + api_token: Option, + is_test: bool, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateSshKeyOutput { + ssh_key_id: u64, + public_key: String, +} + +#[activity(CreateSshKey)] +async fn create_ssh_key( + ctx: &ActivityCtx, + input: &CreateSshKeyInput, +) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + let ns = util::env::namespace(); + + let ssh_key_label = format!("{ns}-{}", input.server_id); + let ssh_key_res = api::create_ssh_key(&client, &ssh_key_label, input.is_test).await?; + + Ok(CreateSshKeyOutput { + ssh_key_id: ssh_key_res.id, + public_key: ssh_key_res.public_key, + }) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateInstanceInput { + api_token: Option, + ssh_public_key: String, + name: String, + datacenter: String, + hardware: String, + tags: Vec, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateInstanceOutput { + linode_id: u64, + server_disk_size: u64, +} + +#[activity(CreateInstance)] +async fn create_instance( + ctx: &ActivityCtx, + input: &CreateInstanceInput, +) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + let create_instance_res = api::create_instance( + &client, + &input.name, + &input.datacenter, + &input.hardware, + &input.tags, + &input.ssh_public_key, + ) + .await?; + let linode_id = create_instance_res.id; + + Ok(CreateInstanceOutput { + linode_id, + server_disk_size: create_instance_res.specs.disk, + }) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct WaitInstanceReadyInput { + api_token: Option, + linode_id: u64, +} + +#[activity(WaitInstanceReady)] +async fn wait_instance_ready( + ctx: &ActivityCtx, + input: &WaitInstanceReadyInput, +) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::wait_instance_ready(&client, input.linode_id).await +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateDisksInput { + api_token: Option, + image: String, + ssh_public_key: String, + linode_id: u64, + disk_size: u64, +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateDisksOutput { + boot_id: u64, + swap_id: u64, +} + +#[activity(CreateDisks)] +async fn create_disks( + ctx: &ActivityCtx, + input: &CreateDisksInput, +) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + let create_disks_res = api::create_disks( + &client, + &input.ssh_public_key, + input.linode_id, + &input.image, + input.disk_size, + ) + .await?; + + Ok(CreateDisksOutput { + boot_id: create_disks_res.boot_id, + swap_id: create_disks_res.swap_id, + }) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateInstanceConfigInput { + api_token: Option, + vlan_ip: Option, + linode_id: u64, + disks: CreateDisksOutput, +} + +#[activity(CreateInstanceConfig)] +async fn create_instance_config( + ctx: &ActivityCtx, + input: &CreateInstanceConfigInput, +) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::create_instance_config( + &client, + input.vlan_ip.as_ref(), + input.linode_id, + input.disks.boot_id, + input.disks.swap_id, + ) + .await +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct CreateFirewallInput { + server_id: Uuid, + api_token: Option, + firewall_preset: FirewallPreset, + tags: Vec, + linode_id: u64, +} + +#[activity(CreateFirewall)] +async fn create_firewall(ctx: &ActivityCtx, input: &CreateFirewallInput) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + let firewall_res = api::create_firewall( + &client, + &input.firewall_preset, + &input.tags, + input.linode_id, + ) + .await?; + + Ok(firewall_res.id) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct BootInstanceInput { + api_token: Option, + linode_id: u64, +} + +#[activity(BootInstance)] +async fn boot_instance(ctx: &ActivityCtx, input: &BootInstanceInput) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::boot_instance(&client, input.linode_id).await?; + + Ok(()) +} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct GetPublicIpInput { + api_token: Option, + linode_id: u64, +} + +#[activity(GetPublicIp)] +async fn get_public_ip(ctx: &ActivityCtx, input: &GetPublicIpInput) -> GlobalResult { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::get_public_ip(&client, input.linode_id).await +} + +#[signal("linode-server-provision-complete")] +pub struct ProvisionComplete { + pub linode_id: u64, + pub public_ip: Ipv4Addr, + pub boot_disk_id: u64, +} + +#[signal("linode-server-provision-failed")] +pub struct ProvisionFailed { + pub err: String, +} + +#[signal("linode-server-destroy")] +pub struct Destroy {} + +#[derive(Debug, Serialize, Deserialize, Hash)] +struct DestroyInstanceInput { + api_token: Option, + linode_id: u64, + ssh_key_id: u64, + firewall_id: u64, +} + +#[activity(DestroyInstance)] +async fn destroy_instance(ctx: &ActivityCtx, input: &DestroyInstanceInput) -> GlobalResult<()> { + // Build HTTP client + let client = client::Client::new(input.api_token.clone()).await?; + + api::delete_instance(&client, input.linode_id).await?; + api::delete_ssh_key(&client, input.ssh_key_id).await?; + api::delete_firewall(&client, input.firewall_id).await?; + + Ok(()) +} diff --git a/svc/pkg/linode/standalone/gc/Cargo.toml b/svc/pkg/linode/standalone/gc/Cargo.toml index 4a2cd1ce5..f3999a289 100644 --- a/svc/pkg/linode/standalone/gc/Cargo.toml +++ b/svc/pkg/linode/standalone/gc/Cargo.toml @@ -7,20 +7,21 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } chrono = "0.4" reqwest = "0.11" rivet-connection = { path = "../../../../../lib/connection" } rivet-health-checks = { path = "../../../../../lib/health-checks" } rivet-metrics = { path = "../../../../../lib/metrics" } -rivet-operation = { path = "../../../../../lib/operation/core" } rivet-runtime = { path = "../../../../../lib/runtime" } serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" tokio = { version = "1.29", features = ["full"] } tracing = "0.1" tracing-subscriber = { version = "0.3", default-features = false, features = ["fmt", "json", "ansi"] } -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } -util-linode = { package = "rivet-util-linode", path = "../../util" } + +cluster = { path = "../../../cluster" } +linode = { path = "../.." } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" @@ -28,4 +29,3 @@ rev = "08d6e61aa0572e7ec557abbedb72cebb96e1ac5b" default-features = false [dev-dependencies] -chirp-worker = { path = "../../../../../lib/chirp/worker" } diff --git a/svc/pkg/linode/standalone/gc/src/lib.rs b/svc/pkg/linode/standalone/gc/src/lib.rs index de7d1279c..57fb524ee 100644 --- a/svc/pkg/linode/standalone/gc/src/lib.rs +++ b/svc/pkg/linode/standalone/gc/src/lib.rs @@ -1,40 +1,27 @@ -use futures_util::{FutureExt, StreamExt, TryStreamExt}; -use proto::backend; +use std::convert::TryInto; + +use chirp_workflow::prelude::*; +use cluster::types::Provider; +use futures_util::{StreamExt, TryStreamExt}; +use linode::util::{api, client}; use reqwest::header; -use rivet_operation::prelude::*; use serde_json::json; -use util_linode::api; - -#[derive(sqlx::FromRow)] -struct LinodePrebakeServer { - install_hash: String, - datacenter_id: Uuid, - pool_type: i64, - - ssh_key_id: i64, - linode_id: Option, - firewall_id: Option, -} #[tracing::instrument(skip_all)] pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("linode-gc"); let cache = rivet_cache::CacheInner::from_env(pools.clone())?; - let ctx = OperationContext::new( - "linode-gc".into(), - std::time::Duration::from_secs(60), + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, rivet_connection::Connection::new(client, pools, cache), - Uuid::new_v4(), - Uuid::new_v4(), - util::timestamp::now(), - util::timestamp::now(), - (), - ); + "linode-gc", + ) + .await?; let dc_rows = sql_fetch_all!( - [ctx, (i64, String,)] + [ctx, (i64, Option>, String,)] " - SELECT DISTINCT provider, provider_api_token + SELECT DISTINCT provider, provider2, provider_api_token FROM db_cluster.datacenters WHERE provider_api_token IS NOT NULL ", @@ -42,7 +29,8 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { .await? .into_iter() .chain(std::iter::once(( - backend::cluster::Provider::Linode as i64, + 0, + Some(sqlx::types::Json(Provider::Linode)), util::env::read_secret(&["linode", "token"]).await?, ))); @@ -56,13 +44,15 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { header::HeaderValue::from_str(&serde_json::to_string(&filter)?)?, ); - for (provider, api_token) in dc_rows { - let provider = unwrap!(backend::cluster::Provider::from_i32(provider as i32)); + for (provider, provider2, api_token) in dc_rows { + let provider = if let Some(provider) = provider2 { + provider.0 + } else { + provider.try_into()? + }; match provider { - backend::cluster::Provider::Linode => { - run_for_linode_account(&ctx, &api_token, &headers).await? - } + Provider::Linode => run_for_linode_account(&ctx, &api_token, &headers).await?, } } @@ -70,146 +60,77 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { } async fn run_for_linode_account( - ctx: &OperationContext<()>, + ctx: &StandaloneCtx, api_token: &str, headers: &header::HeaderMap, ) -> GlobalResult<()> { // Build HTTP client let client = - util_linode::Client::new_with_headers(Some(api_token.to_string()), headers.clone()).await?; + client::Client::new_with_headers(Some(api_token.to_string()), headers.clone()).await?; let complete_images = api::list_custom_images(&client).await?; - delete_expired_images(&client, &complete_images).await?; - - // Get image ids - let image_ids = complete_images - .into_iter() - .map(|x| x.id) - .collect::>(); - if image_ids.len() == util_linode::api::CUSTOM_IMAGE_LIST_SIZE { + if complete_images.len() == linode::util::api::CUSTOM_IMAGE_LIST_SIZE { // We don't need to paginate since we'll never have more than // `number of regions * number of pools * 2` images which is not more than 500 (x2 is for the old + // new images) tracing::warn!("page limit reached, new images may not be returned"); } - let prebake_servers = rivet_pools::utils::crdb::tx(&ctx.crdb().await?, |tx| { - let ctx = ctx.clone(); - let image_ids = image_ids.clone(); + delete_expired_images(ctx, &complete_images).await?; - get_prebake_servers(ctx, tx, image_ids).boxed() - }) - .await?; - - futures_util::stream::iter(prebake_servers.iter()) - .map(|server| { - let client = client.clone(); - - async move { destroy(&client, server).await } - }) - .buffer_unordered(8) - .try_collect::>() - .await?; - - Ok(()) -} + // Get image ids + let image_ids = complete_images + .into_iter() + .map(|x| x.id) + .collect::>(); -#[tracing::instrument(skip_all)] -async fn get_prebake_servers( - ctx: OperationContext<()>, - tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, - image_ids: Vec, -) -> GlobalResult> { - let prebake_servers = sql_fetch_all!( - [ctx, LinodePrebakeServer, @tx tx] + // Set images as complete + let incomplete_images = sql_fetch_all!( + [ctx, (String,)] " - SELECT - install_hash, datacenter_id, pool_type, - ssh_key_id, linode_id, firewall_id - FROM db_cluster.server_images_linode + UPDATE db_linode.server_images + SET complete_ts = $2 + FROM db_linode.server_images WHERE image_id = ANY($1) AND - destroy_ts IS NULL - FOR UPDATE + complete_ts IS NULL + RETURNING image_id ", image_ids, + util::timestamp::now(), ) .await?; - if prebake_servers.is_empty() { - return Ok(Vec::new()); - } - - let primary_keys = prebake_servers - .iter() - .map(|server| { - ( - &server.install_hash, - &server.datacenter_id, - server.pool_type, - ) + futures_util::stream::iter(incomplete_images.into_iter()) + .map(|(image_id,)| { + let ctx = ctx.clone(); + + async move { + ctx.tagged_signal( + &json!({ + "image_id": &image_id, + }), + linode::workflows::image::CreateComplete { image_id }, + ) + .await + } }) - .collect::>(); - let primary_keys = serde_json::to_string(&primary_keys)?; - - // Update image id so it can now be used in provisioning - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.server_images AS i - SET provider_image_id = m.image_id - FROM ( - SELECT - install_hash, datacenter_id, pool_type, image_id - FROM db_cluster.server_images_linode AS s - INNER JOIN jsonb_array_elements($1::JSONB) AS q - ON - s.install_hash = (q->>0)::TEXT AND - s.datacenter_id = (q->>1)::UUID AND - s.pool_type = (q->>2)::INT - WHERE destroy_ts IS NULL - ) AS m - WHERE - i.provider = $2 AND - i.install_hash = m.install_hash AND - i.datacenter_id = m.datacenter_id AND - i.pool_type = m.pool_type - ", - &primary_keys, - backend::cluster::Provider::Linode as i64, - ) - .await?; - - // Remove records - sql_execute!( - [ctx, @tx tx] - " - UPDATE db_cluster.server_images_linode AS s - SET destroy_ts = $2 - FROM jsonb_array_elements($1::JSONB) AS q - WHERE - s.install_hash = (q->>0)::TEXT AND - s.datacenter_id = (q->>1)::UUID AND - s.pool_type = (q->>2)::INT AND - destroy_ts IS NULL - ", - &primary_keys, - util::timestamp::now(), - ) - .await?; + .buffer_unordered(8) + .try_collect::>() + .await?; - Ok(prebake_servers) + Ok(()) } async fn delete_expired_images( - client: &util_linode::Client, + ctx: &StandaloneCtx, complete_images: &[api::CustomImage], ) -> GlobalResult<()> { // Prebake images have an expiration because of their server token. We add 2 days of padding here for // safety let expiration = chrono::Utc::now() - - chrono::Duration::milliseconds(util_cluster::SERVER_TOKEN_TTL) + - chrono::Duration::milliseconds(cluster::util::SERVER_TOKEN_TTL) + chrono::Duration::days(2); let expired_images = complete_images @@ -223,9 +144,17 @@ async fn delete_expired_images( futures_util::stream::iter(expired_images) .map(|img| { - let client = client.clone(); - - async move { api::delete_custom_image(&client, &img.id).await } + let ctx = ctx.clone(); + + async move { + ctx.tagged_signal( + &json!({ + "image_id": img.id, + }), + linode::workflows::image::Destroy {}, + ) + .await + } }) .buffer_unordered(8) .try_collect::>() @@ -233,19 +162,3 @@ async fn delete_expired_images( Ok(()) } - -// NOTE: We do not use `cluster-server-destroy` here because this is a prebake server (only -// `cluster-server-install` works with both) -async fn destroy(client: &util_linode::Client, server: &LinodePrebakeServer) -> GlobalResult<()> { - if let Some(linode_id) = server.linode_id { - api::delete_instance(client, linode_id).await?; - } - - api::delete_ssh_key(client, server.ssh_key_id).await?; - - if let Some(firewall_id) = server.firewall_id { - api::delete_firewall(client, firewall_id).await?; - } - - Ok(()) -} diff --git a/svc/pkg/linode/standalone/gc/src/main.rs b/svc/pkg/linode/standalone/gc/src/main.rs index ba458e376..43207f627 100644 --- a/svc/pkg/linode/standalone/gc/src/main.rs +++ b/svc/pkg/linode/standalone/gc/src/main.rs @@ -1,6 +1,6 @@ use std::time::Duration; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; fn main() -> GlobalResult<()> { rivet_runtime::run(start()).unwrap() diff --git a/svc/pkg/linode/standalone/gc/tests/integration.rs b/svc/pkg/linode/standalone/gc/tests/integration.rs index 150400283..3434577cd 100644 --- a/svc/pkg/linode/standalone/gc/tests/integration.rs +++ b/svc/pkg/linode/standalone/gc/tests/integration.rs @@ -1,5 +1,5 @@ use ::linode_gc::run_from_env; -use chirp_worker::prelude::*; +use chirp_workflow::prelude::*; #[tokio::test(flavor = "multi_thread")] async fn basic() { @@ -12,6 +12,4 @@ async fn basic() { let pools = rivet_pools::from_env("linode-gc-test").await.unwrap(); run_from_env(pools).await.unwrap(); - - // TODO: Check that image_id was set in `server_images` table } diff --git a/svc/pkg/linode/worker/tests/prebake_install_complete.rs b/svc/pkg/linode/tests/image.rs similarity index 100% rename from svc/pkg/linode/worker/tests/prebake_install_complete.rs rename to svc/pkg/linode/tests/image.rs diff --git a/svc/pkg/linode/ops/instance-type-get/tests/integration.rs b/svc/pkg/linode/tests/instance_type_get.rs similarity index 100% rename from svc/pkg/linode/ops/instance-type-get/tests/integration.rs rename to svc/pkg/linode/tests/instance_type_get.rs diff --git a/svc/pkg/linode/ops/server-destroy/tests/integration.rs b/svc/pkg/linode/tests/server_destroy.rs similarity index 100% rename from svc/pkg/linode/ops/server-destroy/tests/integration.rs rename to svc/pkg/linode/tests/server_destroy.rs diff --git a/svc/pkg/linode/ops/server-provision/tests/integration.rs b/svc/pkg/linode/tests/server_provision.rs similarity index 100% rename from svc/pkg/linode/ops/server-provision/tests/integration.rs rename to svc/pkg/linode/tests/server_provision.rs diff --git a/svc/pkg/linode/util/Cargo.toml b/svc/pkg/linode/util/Cargo.toml deleted file mode 100644 index 4f385e77d..000000000 --- a/svc/pkg/linode/util/Cargo.toml +++ /dev/null @@ -1,15 +0,0 @@ -[package] -name = "rivet-util-linode" -version = "0.1.0" -edition = "2021" -authors = ["Rivet Gaming, LLC "] -license = "Apache-2.0" - -[dependencies] -chrono = "0.4" -rand = "0.8" -reqwest = { version = "0.11", features = ["json"] } -rivet-operation = { path = "../../../../lib/operation/core" } -serde = { version = "1.0", features = ["derive"] } -serde_json = "1.0" -ssh-key = "0.6.3" diff --git a/svc/pkg/linode/worker/Cargo.toml b/svc/pkg/linode/worker/Cargo.toml index 4ac5b98f4..ced034a61 100644 --- a/svc/pkg/linode/worker/Cargo.toml +++ b/svc/pkg/linode/worker/Cargo.toml @@ -12,10 +12,9 @@ chirp-worker = { path = "../../../../lib/chirp/worker" } rivet-health-checks = { path = "../../../../lib/health-checks" } rivet-metrics = { path = "../../../../lib/metrics" } rivet-runtime = { path = "../../../../lib/runtime" } -util-cluster = { package = "rivet-util-cluster", path = "../../cluster/util" } util-linode = { package = "rivet-util-linode", path = "../util" } -cluster-datacenter-get = { path = "../../cluster/ops/datacenter-get" } +cluster = { path = "../../cluster" } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" diff --git a/svc/pkg/linode/worker/src/lib.rs b/svc/pkg/linode/worker/src/lib.rs deleted file mode 100644 index 3719b10aa..000000000 --- a/svc/pkg/linode/worker/src/lib.rs +++ /dev/null @@ -1 +0,0 @@ -pub mod workers; diff --git a/svc/pkg/linode/worker/src/workers/mod.rs b/svc/pkg/linode/worker/src/workers/mod.rs deleted file mode 100644 index d54e70dc6..000000000 --- a/svc/pkg/linode/worker/src/workers/mod.rs +++ /dev/null @@ -1,4 +0,0 @@ -pub mod prebake_install_complete; -pub mod prebake_provision; - -chirp_worker::workers![prebake_install_complete, prebake_provision,]; diff --git a/svc/pkg/linode/worker/src/workers/prebake_install_complete.rs b/svc/pkg/linode/worker/src/workers/prebake_install_complete.rs deleted file mode 100644 index 38b0872ff..000000000 --- a/svc/pkg/linode/worker/src/workers/prebake_install_complete.rs +++ /dev/null @@ -1,77 +0,0 @@ -use std::net::IpAddr; - -use chirp_worker::prelude::*; -use proto::backend::pkg::*; -use util_linode::api; - -#[derive(sqlx::FromRow)] -struct PrebakeServer { - install_hash: String, - datacenter_id: Uuid, - pool_type: i64, - - linode_id: i64, - disk_id: i64, -} - -#[worker(name = "linode-prebake-install-complete")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - let public_ip = unwrap!(ctx.public_ip.parse::(), "invalid public ip"); - - let prebake_server = sql_fetch_one!( - [ctx, PrebakeServer] - " - SELECT - install_hash, datacenter_id, pool_type, linode_id, disk_id - FROM db_cluster.server_images_linode - WHERE - public_ip = $1 AND - destroy_ts IS NULL - ", - public_ip, - ) - .await?; - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - - // Build HTTP client - let client = util_linode::Client::new(datacenter.provider_api_token.clone()).await?; - - // Shut down server before creating custom image - api::shut_down(&client, prebake_server.linode_id).await?; - - // NOTE: Linode imposes a restriction of 50 characters on custom image labels, so unfortunately we cannot - // use the image variant as the name. All we need from the label is for it to be unique. Keep in mind that - // the UUID and hyphen take 37 characters, leaving us with 13 for the namespace name - let name = format!("{}-{}", util::env::namespace(), Uuid::new_v4()); - - let create_image_res = api::create_custom_image(&client, &name, prebake_server.disk_id).await?; - - // Write image id - sql_execute!( - [ctx] - " - UPDATE db_cluster.server_images_linode - SET image_id = $4 - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - prebake_server.install_hash, - prebake_server.datacenter_id, - prebake_server.pool_type, - create_image_res.id, - ) - .await?; - - Ok(()) -} diff --git a/svc/pkg/linode/worker/src/workers/prebake_provision.rs b/svc/pkg/linode/worker/src/workers/prebake_provision.rs deleted file mode 100644 index fb834448a..000000000 --- a/svc/pkg/linode/worker/src/workers/prebake_provision.rs +++ /dev/null @@ -1,267 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, cluster::PoolType, pkg::*}; -use util_linode::api; - -#[worker(name = "linode-prebake-provision")] -async fn worker( - ctx: &OperationContext, -) -> GlobalResult<()> { - let crdb = ctx.crdb().await?; - let datacenter_id = unwrap_ref!(ctx.datacenter_id).as_uuid(); - let pool_type = unwrap!(PoolType::from_i32(ctx.pool_type)); - - let datacenter_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: vec![datacenter_id.into()], - }) - .await?; - let datacenter = unwrap!(datacenter_res.datacenters.first()); - - let ns = util::env::namespace(); - let pool_type_str = match pool_type { - PoolType::Job => "job", - PoolType::Gg => "gg", - PoolType::Ats => "ats", - }; - let provider_datacenter_id = &ctx.provider_datacenter_id; - // Prebake server labels just have to be unique, they are ephemeral - let name = format!("{ns}-{}", Uuid::new_v4()); - - let tags = ctx - .tags - .iter() - .cloned() - .chain([ - "prebake".to_string(), - format!("rivet-{ns}"), - format!("{ns}-{provider_datacenter_id}"), - format!("{ns}-{pool_type_str}"), - format!("{ns}-{provider_datacenter_id}-{pool_type_str}"), - ]) - .collect::>(); - - // Build context - let prebake_server = api::ProvisionCtx { - datacenter: provider_datacenter_id.clone(), - name, - hardware: util_linode::consts::PREBAKE_HARDWARE.to_string(), - vlan_ip: None, - tags, - firewall_inbound: vec![util::net::default_firewall()], - }; - - // Build HTTP client - let client = util_linode::Client::new(datacenter.provider_api_token.clone()).await?; - - match provision(ctx, &crdb, &client, datacenter_id, &prebake_server).await { - Ok(public_ip) => { - let request_id = Uuid::new_v4(); - - // Continue to install - msg!([ctx] cluster::msg::server_install(request_id) { - request_id: Some(request_id.into()), - public_ip: public_ip, - pool_type: ctx.pool_type, - server_id: None, - datacenter_id: ctx.datacenter_id, - provider: backend::cluster::Provider::Linode as i32, - initialize_immediately: false, - }) - .await?; - } - // Handle provisioning errors gracefully - Err(err) => { - tracing::error!(?err, "failed to provision server, destroying"); - destroy(ctx, &crdb, &client, datacenter_id).await?; - - // NOTE: This will retry indefinitely to provision a prebake server - retry_bail!("failed to provision server"); - } - } - - Ok(()) -} - -async fn provision( - ctx: &OperationContext, - crdb: &CrdbPool, - client: &util_linode::Client, - datacenter_id: Uuid, - server: &api::ProvisionCtx, -) -> GlobalResult { - let server_id = Uuid::new_v4(); - let ns = util::env::namespace(); - - // Create SSH key - let ssh_key_label = format!("{ns}-{server_id}"); - let ssh_key_res = api::create_ssh_key(client, &ssh_key_label, false).await?; - - // Write SSH key id - sql_execute!( - [ctx] - " - INSERT INTO db_cluster.server_images_linode ( - install_hash, - datacenter_id, - pool_type, - ssh_key_id - ) - VALUES ($1, $2, $3, $4) - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - ssh_key_res.id as i64, - ) - .await?; - - let create_instance_res = api::create_instance(client, server, &ssh_key_res.public_key).await?; - let linode_id = create_instance_res.id; - - // Write linode id - sql_execute!( - [ctx] - " - UPDATE db_cluster.server_images_linode - SET linode_id = $4 - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - linode_id as i64, - ) - .await?; - - api::wait_instance_ready(client, linode_id).await?; - - let create_disks_res = api::create_disks( - client, - &ssh_key_res.public_key, - linode_id, - "linode/debian11", - create_instance_res.specs.disk, - ) - .await?; - - api::create_instance_config(client, server, linode_id, &create_disks_res).await?; - - let firewall_res = api::create_firewall(client, server, linode_id).await?; - - // Write firewall id - sql_execute!( - [ctx, &crdb] - " - UPDATE db_cluster.server_images_linode - SET firewall_id = $4 - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - firewall_res.id as i64, - ) - .await?; - - api::boot_instance(client, linode_id).await?; - - let public_ip = api::get_public_ip(client, linode_id).await?.to_string(); - - // Write SSH key id - sql_execute!( - [ctx, &crdb] - " - UPDATE db_cluster.server_images_linode - SET - disk_id = $4, - public_ip = $5 - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - create_disks_res.boot_id as i64, - &public_ip, - ) - .await?; - - Ok(public_ip) -} - -#[derive(sqlx::FromRow)] -struct LinodeData { - ssh_key_id: i64, - linode_id: Option, - firewall_id: Option, -} - -async fn destroy( - ctx: &OperationContext, - crdb: &CrdbPool, - client: &util_linode::Client, - datacenter_id: Uuid, -) -> GlobalResult<()> { - let data = sql_fetch_optional!( - [ctx, LinodeData, &crdb] - " - SELECT ssh_key_id, linode_id, firewall_id - FROM db_cluster.server_images_linode - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - ) - .await?; - - let Some(data) = data else { - tracing::warn!("deleting server that doesn't exist"); - return Ok(()); - }; - - if let Some(linode_id) = data.linode_id { - api::delete_instance(client, linode_id).await?; - } - - api::delete_ssh_key(client, data.ssh_key_id).await?; - - if let Some(firewall_id) = data.firewall_id { - api::delete_firewall(client, firewall_id).await?; - } - - // Remove record - sql_execute!( - [ctx, &crdb] - " - UPDATE db_cluster.server_images_linode - SET destroy_ts = $4 - WHERE - install_hash = $1 AND - datacenter_id = $2 AND - pool_type = $3 AND - destroy_ts IS NULL - ", - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - ctx.pool_type as i64, - util::timestamp::now(), - ) - .await?; - - Ok(()) -} diff --git a/svc/pkg/linode/worker/tests/prebake_provision.rs b/svc/pkg/linode/worker/tests/prebake_provision.rs deleted file mode 100644 index af922d79f..000000000 --- a/svc/pkg/linode/worker/tests/prebake_provision.rs +++ /dev/null @@ -1,73 +0,0 @@ -use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; - -#[worker_test] -async fn prebake_provision(ctx: TestCtx) { - if !util::feature::server_provision() { - return; - } - - let cluster_id = Uuid::new_v4(); - let datacenter_id = Uuid::new_v4(); - let pool_type = backend::cluster::PoolType::Ats; - let provider_datacenter_id = "us-southeast".to_string(); - - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - display_name: util::faker::ident(), - - provider: backend::cluster::Provider::Linode as i32, - provider_datacenter_id: provider_datacenter_id.clone(), - provider_api_token: None, - - pools: Vec::new(), - - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) - .await - .unwrap(); - - msg!([ctx] linode::msg::prebake_provision(datacenter_id, pool_type as i32) { - datacenter_id: Some(datacenter_id.into()), - pool_type: pool_type as i32, - provider_datacenter_id: provider_datacenter_id, - tags: vec!["test".to_string()], - }) - .await - .unwrap(); - - // Wait for server to have an ip - loop { - tokio::time::sleep(std::time::Duration::from_secs(5)).await; - - let (exists,) = sql_fetch_one!( - [ctx, (bool,)] - " - SELECT EXISTS ( - SELECT 1 - FROM db_cluster.server_images_linode - WHERE - provider = $1 AND - install_hash = $2 AND - datacenter_id = $3 AND - pool_type = $4 AND - public_ip IS NOT NULL AND - destroy_ts IS NULL - ) - ", - backend::cluster::Provider::Linode as i64, - util_cluster::INSTALL_SCRIPT_HASH, - datacenter_id, - pool_type as i64, - ) - .await - .unwrap(); - - if exists { - break; - } - } -} diff --git a/svc/pkg/mm-config/ops/version-prepare/Cargo.toml b/svc/pkg/mm-config/ops/version-prepare/Cargo.toml index 4a28c7df0..150be7535 100644 --- a/svc/pkg/mm-config/ops/version-prepare/Cargo.toml +++ b/svc/pkg/mm-config/ops/version-prepare/Cargo.toml @@ -8,15 +8,16 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } heck = "0.3" +nomad-client = "0.0.9" prost = "0.10" reqwest = "0.11" rivet-operation = { path = "../../../../../lib/operation/core" } s3-util = { path = "../../../../../lib/s3-util" } -nomad-client = "0.0.9" util-build = { package = "rivet-util-build", path = "../../../build/util" } util-job = { package = "rivet-util-job", path = "../../../job/util" } build-get = { path = "../../../build/ops/get" } +cluster = { path = "../../../cluster" } upload-get = { path = "../../../upload/ops/get" } region-get = { path = "../../../region/ops/get" } tier-list = { path = "../../../tier/ops/list" } diff --git a/svc/pkg/mm-config/ops/version-prepare/src/prewarm_ats.rs b/svc/pkg/mm-config/ops/version-prepare/src/prewarm_ats.rs index 894f70c29..4c10c01b1 100644 --- a/svc/pkg/mm-config/ops/version-prepare/src/prewarm_ats.rs +++ b/svc/pkg/mm-config/ops/version-prepare/src/prewarm_ats.rs @@ -57,14 +57,14 @@ pub async fn prewarm_ats_cache( FROM db_cluster.servers WHERE datacenter_id = ANY($1) AND - pool_type = $2 AND + pool_type2 = $2 AND vlan_ip IS NOT NULL AND drain_ts IS NULL AND cloud_destroy_ts IS NULL ", // NOTE: region_id is just the old name for datacenter_id prewarm_ctx.region_ids.iter().cloned().collect::>(), - backend::cluster::PoolType::Ats as i64, + serde_json::to_string(&cluster::types::PoolType::Ats)?, ) .await?; diff --git a/svc/pkg/mm/worker/Cargo.toml b/svc/pkg/mm/worker/Cargo.toml index 1097a4fa0..aeaa3111a 100644 --- a/svc/pkg/mm/worker/Cargo.toml +++ b/svc/pkg/mm/worker/Cargo.toml @@ -29,6 +29,7 @@ util-job = { package = "rivet-util-job", path = "../../job/util" } util-mm = { package = "rivet-util-mm", path = "../util" } build-get = { path = "../../build/ops/get" } +cluster = { path = "../../cluster" } game-get = { path = "../../game/ops/get" } game-namespace-get = { path = "../../game/ops/namespace-get" } game-version-get = { path = "../../game/ops/version-get" } diff --git a/svc/pkg/mm/worker/src/workers/lobby_create/mod.rs b/svc/pkg/mm/worker/src/workers/lobby_create/mod.rs index 7042899d5..3bac84cf8 100644 --- a/svc/pkg/mm/worker/src/workers/lobby_create/mod.rs +++ b/svc/pkg/mm/worker/src/workers/lobby_create/mod.rs @@ -809,7 +809,7 @@ async fn resolve_job_runner_binary_url( FROM db_cluster.servers WHERE datacenter_id = $1 AND - pool_type = $2 AND + pool_type2 = $2 AND vlan_ip IS NOT NULL AND install_complete_ts IS NOT NULL AND drain_ts IS NULL AND @@ -822,7 +822,7 @@ async fn resolve_job_runner_binary_url( ", // NOTE: region_id is just the old name for datacenter_id ®ion_id, - backend::cluster::PoolType::Ats as i64, + serde_json::to_string(&cluster::types::PoolType::Ats)?, ) .await?; @@ -936,7 +936,7 @@ async fn resolve_image_artifact_url( FROM db_cluster.servers WHERE datacenter_id = $1 AND - pool_type = $2 AND + pool_type2 = $2 AND vlan_ip IS NOT NULL AND install_complete_ts IS NOT NULL AND drain_ts IS NULL AND @@ -950,7 +950,7 @@ async fn resolve_image_artifact_url( ", // NOTE: region_id is just the old name for datacenter_id ®ion_id, - backend::cluster::PoolType::Ats as i64, + serde_json::to_string(&cluster::types::PoolType::Ats)?, hash, ) .await?; diff --git a/svc/pkg/monolith/standalone/worker/Cargo.toml b/svc/pkg/monolith/standalone/worker/Cargo.toml index 047dd4ead..0c3dc3897 100644 --- a/svc/pkg/monolith/standalone/worker/Cargo.toml +++ b/svc/pkg/monolith/standalone/worker/Cargo.toml @@ -23,13 +23,11 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ cdn-worker = { path = "../../../cdn/worker" } cf-custom-hostname-worker = { path = "../../../cf-custom-hostname/worker" } cloud-worker = { path = "../../../cloud/worker" } -cluster-worker = { path = "../../../cluster/worker" } external-worker = { path = "../../../external/worker" } game-user-worker = { path = "../../../game-user/worker" } job-log-worker = { path = "../../../job-log/worker" } job-run-worker = { path = "../../../job-run/worker" } kv-worker = { path = "../../../kv/worker" } -linode-worker = { path = "../../../linode/worker" } mm-worker = { path = "../../../mm/worker" } team-invite-worker = { path = "../../../team-invite/worker" } team-worker = { path = "../../../team/worker" } diff --git a/svc/pkg/monolith/standalone/worker/src/lib.rs b/svc/pkg/monolith/standalone/worker/src/lib.rs index 4cb67dfb0..98df9333b 100644 --- a/svc/pkg/monolith/standalone/worker/src/lib.rs +++ b/svc/pkg/monolith/standalone/worker/src/lib.rs @@ -25,13 +25,11 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { cdn_worker, cf_custom_hostname_worker, cloud_worker, - cluster_worker, external_worker, game_user_worker, job_log_worker, job_run_worker, kv_worker, - linode_worker, mm_worker, team_invite_worker, team_worker, diff --git a/svc/pkg/monolith/standalone/workflow-worker/Cargo.toml b/svc/pkg/monolith/standalone/workflow-worker/Cargo.toml index ff6ef48e9..47e8b3918 100644 --- a/svc/pkg/monolith/standalone/workflow-worker/Cargo.toml +++ b/svc/pkg/monolith/standalone/workflow-worker/Cargo.toml @@ -10,3 +10,6 @@ chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } rivet-health-checks = { path = "../../../../../lib/health-checks" } rivet-metrics = { path = "../../../../../lib/metrics" } rivet-runtime = { path = "../../../../../lib/runtime" } + +cluster = { path = "../../../cluster" } +linode = { path = "../../../linode" } diff --git a/svc/pkg/monolith/standalone/workflow-worker/src/lib.rs b/svc/pkg/monolith/standalone/workflow-worker/src/lib.rs index a3efab952..c3b90a2fe 100644 --- a/svc/pkg/monolith/standalone/workflow-worker/src/lib.rs +++ b/svc/pkg/monolith/standalone/workflow-worker/src/lib.rs @@ -2,7 +2,7 @@ use chirp_workflow::prelude::*; #[tracing::instrument(skip_all)] pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { - let reg = Registry::new(); + let reg = cluster::registry().merge(linode::registry()); let db = db::DatabasePostgres::from_pool(pools.crdb().unwrap()); let worker = Worker::new(reg.handle(), db.clone()); diff --git a/svc/pkg/nomad/standalone/monitor/Cargo.toml b/svc/pkg/nomad/standalone/monitor/Cargo.toml index 9f1076584..09378e188 100644 --- a/svc/pkg/nomad/standalone/monitor/Cargo.toml +++ b/svc/pkg/nomad/standalone/monitor/Cargo.toml @@ -7,6 +7,7 @@ license = "Apache-2.0" [dependencies] chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } chrono = "0.4" futures-util = "0.3" indoc = "1.0" @@ -30,6 +31,8 @@ tracing-subscriber = { version = "0.3", default-features = false, features = [ ] } util-job = { package = "rivet-util-job", path = "../../../job/util" } +cluster = { path = "../../../cluster" } + [dependencies.nomad_client] git = "https://github.com/rivet-gg/nomad-client" rev = "abb66bf0c30c7ff5b0c695dae952481c33e538b5" # pragma: allowlist secret diff --git a/svc/pkg/nomad/standalone/monitor/src/lib.rs b/svc/pkg/nomad/standalone/monitor/src/lib.rs index c17e5eefa..25b4663d2 100644 --- a/svc/pkg/nomad/standalone/monitor/src/lib.rs +++ b/svc/pkg/nomad/standalone/monitor/src/lib.rs @@ -1,13 +1,18 @@ -use std::sync::Arc; - -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; mod monitors; use monitors::*; pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { - let shared_client = chirp_client::SharedClient::from_env(pools.clone())?; + let client = chirp_client::SharedClient::from_env(pools.clone())?.wrap_new("nomad-monitor"); + let cache = rivet_cache::CacheInner::from_env(pools.clone())?; let redis_job = pools.redis("persistent")?; + let ctx = StandaloneCtx::new( + chirp_workflow::compat::db_from_pools(&pools).await?, + rivet_connection::Connection::new(client, pools, cache), + "nomad-monitor", + ) + .await?; // Start nomad event monitor let redis_index_key = "nomad:monitor_index"; @@ -18,27 +23,24 @@ pub async fn run_from_env(pools: rivet_pools::Pools) -> GlobalResult<()> { redis_job, redis_index_key, &["Allocation", "Evaluation", "Node"], - |event| handle(shared_client.clone(), event), + |event| handle(ctx.clone(), event), ) .await?; Ok(()) } -async fn handle( - shared_client: Arc, - event: nomad_util::monitor::NomadEvent, -) { +async fn handle(ctx: StandaloneCtx, event: nomad_util::monitor::NomadEvent) { // TODO: Figure out how to abstract the branches if let Some(payload) = event .decode::("Allocation", "PlanResult") .unwrap() { - let client = shared_client.wrap_new("nomad-alloc-plan-monitor"); + // let client = shared_client.wrap_new("nomad-alloc-plan-monitor"); let spawn_res = tokio::task::Builder::new() .name("nomad_alloc_plan_monitor::handle_event") .spawn(async move { - match alloc_plan::handle(client, &payload, event.payload.to_string()).await { + match alloc_plan::handle(ctx, &payload, event.payload.to_string()).await { Ok(_) => {} Err(err) => { tracing::error!(?err, ?payload, "error handling event"); @@ -52,11 +54,11 @@ async fn handle( .decode::("Allocation", "AllocationUpdated") .unwrap() { - let client = shared_client.wrap_new("nomad-alloc-updated-monitor"); + // let client = shared_client.wrap_new("nomad-alloc-updated-monitor"); let spawn_res = tokio::task::Builder::new() .name("nomad_alloc_update_monitor::handle_event") .spawn(async move { - match alloc_update::handle(client, &payload, event.payload.to_string()).await { + match alloc_update::handle(ctx, &payload, event.payload.to_string()).await { Ok(_) => {} Err(err) => { tracing::error!(?err, ?payload, "error handling event"); @@ -70,11 +72,11 @@ async fn handle( .decode::("Evaluation", "EvaluationUpdated") .unwrap() { - let client = shared_client.wrap_new("nomad-eval-update-monitor"); + // let client = shared_client.wrap_new("nomad-eval-update-monitor"); let spawn_res = tokio::task::Builder::new() .name("nomad_eval_update_monitor::handle_event") .spawn(async move { - match eval_update::handle(client, &payload, event.payload.to_string()).await { + match eval_update::handle(ctx, &payload, event.payload.to_string()).await { Ok(_) => {} Err(err) => { tracing::error!(?err, ?payload, "error handling event"); @@ -88,11 +90,11 @@ async fn handle( .decode::("Node", "NodeRegistration") .unwrap() { - let client = shared_client.wrap_new("nomad-node-registration-monitor"); + // let client = shared_client.wrap_new("nomad-node-registration-monitor"); let spawn_res = tokio::task::Builder::new() .name("nomad_node_registration_monitor::handle") .spawn(async move { - match node_registration::handle(client, &payload).await { + match node_registration::handle(ctx, &payload).await { Ok(_) => {} Err(err) => { tracing::error!(?err, ?payload, "error handling event"); @@ -106,11 +108,11 @@ async fn handle( .decode::("Node", "NodeDrain") .unwrap() { - let client = shared_client.wrap_new("nomad-node-drain-monitor"); + // let client = shared_client.wrap_new("nomad-node-drain-monitor"); let spawn_res = tokio::task::Builder::new() .name("nomad_node_drain_monitor::handle") .spawn(async move { - match node_drain::handle(client, &payload).await { + match node_drain::handle(ctx, &payload).await { Ok(_) => {} Err(err) => { tracing::error!(?err, ?payload, "error handling event"); diff --git a/svc/pkg/nomad/standalone/monitor/src/main.rs b/svc/pkg/nomad/standalone/monitor/src/main.rs index 8dada711e..fae749a15 100644 --- a/svc/pkg/nomad/standalone/monitor/src/main.rs +++ b/svc/pkg/nomad/standalone/monitor/src/main.rs @@ -1,4 +1,4 @@ -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; fn main() -> GlobalResult<()> { rivet_runtime::run(start()).unwrap() diff --git a/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_plan.rs b/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_plan.rs index 538835ea8..f0b33f877 100644 --- a/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_plan.rs +++ b/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_plan.rs @@ -1,5 +1,5 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; +use rivet_operation::prelude::proto::backend::pkg::*; use serde::Deserialize; #[derive(Debug, Deserialize)] @@ -9,7 +9,7 @@ pub struct PlanResult { } pub async fn handle( - client: chirp_client::Client, + ctx: StandaloneCtx, PlanResult { allocation: alloc }: &PlanResult, payload_json: String, ) -> GlobalResult<()> { @@ -20,7 +20,7 @@ pub async fn handle( return Ok(()); } - msg!([client] nomad::msg::monitor_alloc_plan(job_id) { + msg!([ctx] nomad::msg::monitor_alloc_plan(job_id) { dispatched_job_id: job_id.clone(), payload_json: payload_json, }) diff --git a/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_update.rs b/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_update.rs index d61bdbfbe..86a296354 100644 --- a/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_update.rs +++ b/svc/pkg/nomad/standalone/monitor/src/monitors/alloc_update.rs @@ -1,5 +1,5 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; +use rivet_operation::prelude::proto::backend::pkg::*; use serde::Deserialize; #[derive(Debug, Clone, Deserialize)] @@ -9,7 +9,7 @@ pub struct AllocationUpdated { } pub async fn handle( - client: chirp_client::Client, + ctx: StandaloneCtx, AllocationUpdated { allocation: alloc }: &AllocationUpdated, payload_json: String, ) -> GlobalResult<()> { @@ -20,7 +20,7 @@ pub async fn handle( return Ok(()); } - msg!([client] nomad::msg::monitor_alloc_update(job_id) { + msg!([ctx] nomad::msg::monitor_alloc_update(job_id) { dispatched_job_id: job_id.clone(), payload_json: payload_json, }) diff --git a/svc/pkg/nomad/standalone/monitor/src/monitors/eval_update.rs b/svc/pkg/nomad/standalone/monitor/src/monitors/eval_update.rs index 4fbcd6e3d..c4bcea12f 100644 --- a/svc/pkg/nomad/standalone/monitor/src/monitors/eval_update.rs +++ b/svc/pkg/nomad/standalone/monitor/src/monitors/eval_update.rs @@ -1,5 +1,5 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; +use rivet_operation::prelude::proto::backend::pkg::*; use serde::Deserialize; #[derive(Debug, Deserialize)] @@ -9,7 +9,7 @@ pub struct PlanResult { } pub async fn handle( - client: chirp_client::Client, + ctx: StandaloneCtx, PlanResult { evaluation: eval }: &PlanResult, payload_json: String, ) -> GlobalResult<()> { @@ -33,7 +33,7 @@ pub async fn handle( return Ok(()); } - msg!([client] nomad::msg::monitor_eval_update(job_id) { + msg!([ctx] nomad::msg::monitor_eval_update(job_id) { dispatched_job_id: job_id.clone(), payload_json: payload_json, }) diff --git a/svc/pkg/nomad/standalone/monitor/src/monitors/node_drain.rs b/svc/pkg/nomad/standalone/monitor/src/monitors/node_drain.rs index 33cc6b1c3..d2ec5044f 100644 --- a/svc/pkg/nomad/standalone/monitor/src/monitors/node_drain.rs +++ b/svc/pkg/nomad/standalone/monitor/src/monitors/node_drain.rs @@ -1,6 +1,6 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; use serde::Deserialize; +use serde_json::json; #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "PascalCase")] @@ -8,10 +8,7 @@ pub struct NodeDrain { node: nomad_client::models::Node, } -pub async fn handle( - client: chirp_client::Client, - NodeDrain { node }: &NodeDrain, -) -> GlobalResult<()> { +pub async fn handle(ctx: StandaloneCtx, NodeDrain { node }: &NodeDrain) -> GlobalResult<()> { let node_id = unwrap_ref!(node.ID); let meta = unwrap_ref!(node.meta, "no metadata on node"); let server_id = util::uuid::parse(unwrap!(meta.get("server-id"), "no server-id in metadata"))?; @@ -27,10 +24,12 @@ pub async fn handle( .unwrap_or_default(); if is_last_drain_complete_message { - msg!([client] nomad::msg::monitor_node_drain_complete(server_id) { - server_id: Some(server_id.into()), - node_id: node_id.to_owned(), - }) + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + cluster::workflows::server::NomadDrainComplete {}, + ) .await?; } } diff --git a/svc/pkg/nomad/standalone/monitor/src/monitors/node_registration.rs b/svc/pkg/nomad/standalone/monitor/src/monitors/node_registration.rs index fe095faab..fc723c084 100644 --- a/svc/pkg/nomad/standalone/monitor/src/monitors/node_registration.rs +++ b/svc/pkg/nomad/standalone/monitor/src/monitors/node_registration.rs @@ -1,6 +1,6 @@ -use proto::backend::pkg::*; -use rivet_operation::prelude::*; +use chirp_workflow::prelude::*; use serde::Deserialize; +use serde_json::json; #[derive(Debug, Clone, Deserialize)] #[serde(rename_all = "PascalCase")] @@ -9,17 +9,21 @@ pub struct NodeRegistration { } pub async fn handle( - client: chirp_client::Client, + ctx: StandaloneCtx, NodeRegistration { node }: &NodeRegistration, ) -> GlobalResult<()> { let node_id = unwrap_ref!(node.ID); let meta = unwrap_ref!(node.meta, "no metadata on node"); let server_id = util::uuid::parse(unwrap!(meta.get("server-id"), "no server-id in metadata"))?; - msg!([client] nomad::msg::monitor_node_registered(server_id) { - server_id: Some(server_id.into()), - node_id: node_id.to_owned(), - }) + ctx.tagged_signal( + &json!({ + "server_id": server_id, + }), + cluster::workflows::server::NomadRegistered { + node_id: node_id.to_owned(), + }, + ) .await?; Ok(()) diff --git a/svc/pkg/region/ops/get/Cargo.toml b/svc/pkg/region/ops/get/Cargo.toml index 2520092cc..c33378c7e 100644 --- a/svc/pkg/region/ops/get/Cargo.toml +++ b/svc/pkg/region/ops/get/Cargo.toml @@ -6,12 +6,12 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } prost = "0.10" +rivet-operation = { path = "../../../../../lib/operation/core" } -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } -cluster-datacenter-location-get = { path = "../../../cluster/ops/datacenter-location-get" } +cluster = { path = "../../../cluster" } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" diff --git a/svc/pkg/region/ops/get/src/lib.rs b/svc/pkg/region/ops/get/src/lib.rs index bff9efdc1..3ec281bbd 100644 --- a/svc/pkg/region/ops/get/src/lib.rs +++ b/svc/pkg/region/ops/get/src/lib.rs @@ -2,38 +2,45 @@ use proto::backend::{self, pkg::*}; use rivet_operation::prelude::*; fn convert_datacenter( - datacenter: &backend::cluster::Datacenter, - locations: &[cluster::datacenter_location_get::response::Datacenter], + datacenter: &cluster::types::Datacenter, + locations: &[cluster::ops::datacenter::location_get::Datacenter], ) -> GlobalResult { - let datacenter_id = unwrap_ref!(datacenter.datacenter_id).as_uuid(); - let provider = unwrap!(backend::cluster::Provider::from_i32(datacenter.provider)); - let coords = locations .iter() .find(|location| location.datacenter_id == datacenter.datacenter_id) - .and_then(|dc| dc.coords.clone()) - .unwrap_or(backend::net::Coordinates { + .map(|dc| dc.coords.clone()) + .unwrap_or(cluster::ops::datacenter::location_get::Coordinates { latitude: 0.0, longitude: 0.0, }); Ok(backend::region::Region { - region_id: datacenter.datacenter_id, + region_id: Some(datacenter.datacenter_id.into()), enabled: true, nomad_region: "global".into(), - nomad_datacenter: datacenter_id.to_string(), - provider: match provider { - backend::cluster::Provider::Linode => "linode".to_string(), + nomad_datacenter: datacenter.datacenter_id.to_string(), + provider: match datacenter.provider { + cluster::types::Provider::Linode => "linode".to_string(), }, provider_region: datacenter.provider_datacenter_id.clone(), - provider_display_name: match provider { - backend::cluster::Provider::Linode => "Linode".to_string(), + provider_display_name: match datacenter.provider { + cluster::types::Provider::Linode => "Linode".to_string(), }, region_display_name: datacenter.display_name.clone(), name_id: datacenter.name_id.clone(), - coords: Some(coords), + coords: Some(backend::net::Coordinates { + latitude: coords.latitude, + longitude: coords.longitude, + }), - build_delivery_method: datacenter.build_delivery_method, + build_delivery_method: match datacenter.build_delivery_method { + cluster::types::BuildDeliveryMethod::TrafficServer => { + backend::cluster::BuildDeliveryMethod::TrafficServer as i32 + } + cluster::types::BuildDeliveryMethod::S3Direct => { + backend::cluster::BuildDeliveryMethod::S3Direct as i32 + } + }, }) } @@ -41,13 +48,24 @@ fn convert_datacenter( async fn handle( ctx: OperationContext, ) -> GlobalResult { + let datacenter_ids = ctx + .region_ids + .iter() + .map(|id| id.as_uuid()) + .collect::>(); let (datacenters_res, locations_res) = tokio::try_join!( - op!([ctx] cluster_datacenter_get { - datacenter_ids: ctx.region_ids.clone(), - }), - op!([ctx] cluster_datacenter_location_get { - datacenter_ids: ctx.region_ids.clone(), - }), + chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::get::Input { + datacenter_ids: datacenter_ids.clone(), + }, + ), + chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::location_get::Input { + datacenter_ids: datacenter_ids.clone(), + }, + ), )?; let regions = datacenters_res diff --git a/svc/pkg/region/ops/list-for-game/Cargo.toml b/svc/pkg/region/ops/list-for-game/Cargo.toml index a7b72c1dc..bb75b7f37 100644 --- a/svc/pkg/region/ops/list-for-game/Cargo.toml +++ b/svc/pkg/region/ops/list-for-game/Cargo.toml @@ -6,12 +6,12 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } prost = "0.10" +rivet-operation = { path = "../../../../../lib/operation/core" } -cluster-get-for-game = { path = "../../../cluster/ops/get-for-game" } -cluster-datacenter-list = { path = "../../../cluster/ops/datacenter-list" } +cluster = { path = "../../../cluster" } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" diff --git a/svc/pkg/region/ops/list-for-game/src/lib.rs b/svc/pkg/region/ops/list-for-game/src/lib.rs index 3b41d46a3..87f956d46 100644 --- a/svc/pkg/region/ops/list-for-game/src/lib.rs +++ b/svc/pkg/region/ops/list-for-game/src/lib.rs @@ -5,24 +5,36 @@ use rivet_operation::prelude::*; async fn handle( ctx: OperationContext, ) -> GlobalResult { - let clusters_res = op!([ctx] cluster_get_for_game { - game_ids: ctx.game_ids.clone(), - }) + let clusters_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::get_for_game::Input { + game_ids: ctx + .game_ids + .iter() + .map(|id| id.as_uuid()) + .collect::>(), + }, + ) .await?; - let datacenter_list_res = op!([ctx] cluster_datacenter_list { - cluster_ids: clusters_res - .games - .iter() - .map(|game| Ok(unwrap!(game.cluster_id))) - .collect::>>()?, - }) + let datacenter_list_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::list::Input { + cluster_ids: clusters_res + .games + .iter() + .map(|game| game.cluster_id) + .collect::>(), + }, + ) .await?; + let datacenter_ids = datacenter_list_res .clusters .iter() .flat_map(|cluster| &cluster.datacenter_ids) .cloned() + .map(Into::into) .collect::>(); Ok(region::list_for_game::Response { diff --git a/svc/pkg/region/ops/list-for-game/tests/integration.rs b/svc/pkg/region/ops/list-for-game/tests/integration.rs index d2067269b..b68a0106a 100644 --- a/svc/pkg/region/ops/list-for-game/tests/integration.rs +++ b/svc/pkg/region/ops/list-for-game/tests/integration.rs @@ -1,15 +1,18 @@ use chirp_worker::prelude::*; -use proto::backend::{self, pkg::*}; +use serde_json::json; #[worker_test] async fn empty(ctx: TestCtx) { let (cluster_id, datacenter_id) = create_dc(&ctx).await; let game_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::game_link(game_id, cluster_id) -> cluster::msg::game_link_complete { - game_id: Some(game_id.into()), - cluster_id: Some(cluster_id.into()), - }) + chirp_workflow::compat::tagged_signal( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::GameLink { game_id }, + ) .await .unwrap(); @@ -31,31 +34,53 @@ async fn create_dc(ctx: &TestCtx) -> (Uuid, Uuid) { let datacenter_id = Uuid::new_v4(); let cluster_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) + chirp_workflow::compat::dispatch_tagged_workflow( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) .await .unwrap(); - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - display_name: util::faker::ident(), + let mut create_sub = + chirp_workflow::compat::subscribe::( + ctx.op_ctx(), + &json!({ + "datacenter_id": datacenter_id, + }), + ) + .await + .unwrap(); + chirp_workflow::compat::tagged_signal( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id, + name_id: util::faker::ident(), + display_name: util::faker::ident(), - provider: backend::cluster::Provider::Linode as i32, - provider_datacenter_id: "us-southeast".to_string(), - provider_api_token: None, + provider: cluster::types::Provider::Linode, + provider_datacenter_id: "us-southeast".to_string(), + provider_api_token: None, - pools: Vec::new(), + pools: Vec::new(), - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) + build_delivery_method: cluster::types::BuildDeliveryMethod::TrafficServer, + prebakes_enabled: false, + }, + ) .await .unwrap(); + create_sub.next().await.unwrap(); + (cluster_id, datacenter_id) } diff --git a/svc/pkg/region/ops/list/Cargo.toml b/svc/pkg/region/ops/list/Cargo.toml index 7be1a8d5a..fb1c6ab69 100644 --- a/svc/pkg/region/ops/list/Cargo.toml +++ b/svc/pkg/region/ops/list/Cargo.toml @@ -6,12 +6,12 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } prost = "0.10" -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } +rivet-operation = { path = "../../../../../lib/operation/core" } -cluster-datacenter-list = { path = "../../../cluster/ops/datacenter-list" } +cluster = { path = "../../../cluster" } [dependencies.sqlx] git = "https://github.com/rivet-gg/sqlx" diff --git a/svc/pkg/region/ops/list/src/lib.rs b/svc/pkg/region/ops/list/src/lib.rs index 59f1bad5d..79c557fdb 100644 --- a/svc/pkg/region/ops/list/src/lib.rs +++ b/svc/pkg/region/ops/list/src/lib.rs @@ -5,9 +5,12 @@ use rivet_operation::prelude::*; async fn handle( ctx: OperationContext, ) -> GlobalResult { - let datacenter_list_res = op!([ctx] cluster_datacenter_list { - cluster_ids: vec![util_cluster::default_cluster_id().into()], - }) + let datacenter_list_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::list::Input { + cluster_ids: vec![cluster::util::default_cluster_id()], + }, + ) .await?; let cluster = unwrap!( datacenter_list_res.clusters.first(), @@ -15,6 +18,11 @@ async fn handle( ); Ok(region::list::Response { - region_ids: cluster.datacenter_ids.clone(), + region_ids: cluster + .datacenter_ids + .iter() + .cloned() + .map(Into::into) + .collect(), }) } diff --git a/svc/pkg/region/ops/resolve-for-game/Cargo.toml b/svc/pkg/region/ops/resolve-for-game/Cargo.toml index 59548df97..3274638f3 100644 --- a/svc/pkg/region/ops/resolve-for-game/Cargo.toml +++ b/svc/pkg/region/ops/resolve-for-game/Cargo.toml @@ -6,11 +6,12 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } +rivet-operation = { path = "../../../../../lib/operation/core" } prost = "0.10" -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } +cluster = { path = "../../../cluster" } region-list-for-game = { path = "../list-for-game" } [dependencies.sqlx] diff --git a/svc/pkg/region/ops/resolve-for-game/src/lib.rs b/svc/pkg/region/ops/resolve-for-game/src/lib.rs index ce883dc35..9fd9ea88e 100644 --- a/svc/pkg/region/ops/resolve-for-game/src/lib.rs +++ b/svc/pkg/region/ops/resolve-for-game/src/lib.rs @@ -12,9 +12,16 @@ async fn handle( }) .await?; - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: region_list_res.region_ids.clone(), - }) + let datacenters_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::get::Input { + datacenter_ids: region_list_res + .region_ids + .iter() + .map(|id| id.as_uuid()) + .collect(), + }, + ) .await?; let regions = datacenters_res @@ -22,7 +29,7 @@ async fn handle( .iter() .filter(|dc| ctx.name_ids.contains(&dc.name_id)) .map(|dc| region::resolve_for_game::response::Region { - region_id: dc.datacenter_id, + region_id: Some(dc.datacenter_id.into()), name_id: dc.name_id.clone(), }) .collect::>(); diff --git a/svc/pkg/region/ops/resolve-for-game/tests/integration.rs b/svc/pkg/region/ops/resolve-for-game/tests/integration.rs index d9c548414..7cd96af21 100644 --- a/svc/pkg/region/ops/resolve-for-game/tests/integration.rs +++ b/svc/pkg/region/ops/resolve-for-game/tests/integration.rs @@ -1,15 +1,19 @@ use chirp_worker::prelude::*; use proto::backend::{self, pkg::*}; +use serde_json::json; #[worker_test] async fn empty(ctx: TestCtx) { let (cluster_id, datacenter_id, dc_name_id) = create_dc(&ctx).await; let game_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::game_link(game_id, cluster_id) -> cluster::msg::game_link_complete { - game_id: Some(game_id.into()), - cluster_id: Some(cluster_id.into()), - }) + chirp_workflow::compat::tagged_signal( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::GameLink { game_id }, + ) .await .unwrap(); @@ -39,31 +43,53 @@ async fn create_dc(ctx: &TestCtx) -> (Uuid, Uuid, String) { let dc_name_id = util::faker::ident(); let cluster_id = Uuid::new_v4(); - msg!([ctx] cluster::msg::create(cluster_id) -> cluster::msg::create_complete { - cluster_id: Some(cluster_id.into()), - name_id: util::faker::ident(), - owner_team_id: None, - }) + chirp_workflow::compat::dispatch_tagged_workflow( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::Input { + cluster_id, + name_id: util::faker::ident(), + owner_team_id: None, + }, + ) .await .unwrap(); - msg!([ctx] cluster::msg::datacenter_create(datacenter_id) -> cluster::msg::datacenter_scale { - datacenter_id: Some(datacenter_id.into()), - cluster_id: Some(cluster_id.into()), - name_id: dc_name_id.clone(), - display_name: util::faker::ident(), + let mut create_sub = + chirp_workflow::compat::subscribe::( + ctx.op_ctx(), + &json!({ + "datacenter_id": datacenter_id, + }), + ) + .await + .unwrap(); + chirp_workflow::compat::tagged_signal( + ctx.op_ctx(), + &json!({ + "cluster_id": cluster_id, + }), + cluster::workflows::cluster::DatacenterCreate { + datacenter_id, + name_id: dc_name_id.clone(), + display_name: util::faker::ident(), - provider: backend::cluster::Provider::Linode as i32, - provider_datacenter_id: "us-southeast".to_string(), - provider_api_token: None, + provider: cluster::types::Provider::Linode, + provider_datacenter_id: "us-southeast".to_string(), + provider_api_token: None, - pools: Vec::new(), + pools: Vec::new(), - build_delivery_method: backend::cluster::BuildDeliveryMethod::TrafficServer as i32, - prebakes_enabled: false, - }) + build_delivery_method: cluster::types::BuildDeliveryMethod::TrafficServer, + prebakes_enabled: false, + }, + ) .await .unwrap(); + create_sub.next().await.unwrap(); + (cluster_id, datacenter_id, dc_name_id) } diff --git a/svc/pkg/region/ops/resolve/Cargo.toml b/svc/pkg/region/ops/resolve/Cargo.toml index 6dbcda2f7..593673733 100644 --- a/svc/pkg/region/ops/resolve/Cargo.toml +++ b/svc/pkg/region/ops/resolve/Cargo.toml @@ -6,11 +6,12 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } +rivet-operation = { path = "../../../../../lib/operation/core" } prost = "0.10" -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } +cluster = { path = "../../../cluster" } region-list = { path = "../list" } [dependencies.sqlx] diff --git a/svc/pkg/region/ops/resolve/src/lib.rs b/svc/pkg/region/ops/resolve/src/lib.rs index f1525aeb0..816ebbd34 100644 --- a/svc/pkg/region/ops/resolve/src/lib.rs +++ b/svc/pkg/region/ops/resolve/src/lib.rs @@ -7,9 +7,16 @@ async fn handle( ) -> GlobalResult { let region_list_res = op!([ctx] region_list { }).await?; - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: region_list_res.region_ids.clone(), - }) + let datacenters_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::get::Input { + datacenter_ids: region_list_res + .region_ids + .iter() + .map(|id| id.as_uuid()) + .collect(), + }, + ) .await?; let regions = datacenters_res @@ -17,7 +24,7 @@ async fn handle( .iter() .filter(|dc| ctx.name_ids.contains(&dc.name_id)) .map(|dc| region::resolve::response::Region { - region_id: dc.datacenter_id, + region_id: Some(dc.datacenter_id.into()), name_id: dc.name_id.clone(), }) .collect::>(); diff --git a/svc/pkg/tier/ops/list/Cargo.toml b/svc/pkg/tier/ops/list/Cargo.toml index c335edf0b..e756c63bb 100644 --- a/svc/pkg/tier/ops/list/Cargo.toml +++ b/svc/pkg/tier/ops/list/Cargo.toml @@ -6,15 +6,13 @@ authors = ["Rivet Gaming, LLC "] license = "Apache-2.0" [dependencies] -rivet-operation = { path = "../../../../../lib/operation/core" } chirp-client = { path = "../../../../../lib/chirp/client" } +chirp-workflow = { path = "../../../../../lib/chirp-workflow/core" } prost = "0.10" -util-cluster = { package = "rivet-util-cluster", path = "../../../cluster/util" } +rivet-operation = { path = "../../../../../lib/operation/core" } -cluster-datacenter-get = { path = "../../../cluster/ops/datacenter-get" } -linode-instance-type-get = { path = "../../../linode/ops/instance-type-get" } +cluster = { path = "../../../cluster" } +linode = { path = "../../../linode" } [dev-dependencies] chirp-worker = { path = "../../../../../lib/chirp/worker" } - -cluster-datacenter-list = { path = "../../../cluster/ops/datacenter-list" } diff --git a/svc/pkg/tier/ops/list/src/lib.rs b/svc/pkg/tier/ops/list/src/lib.rs index ed8638272..89114ac0e 100644 --- a/svc/pkg/tier/ops/list/src/lib.rs +++ b/svc/pkg/tier/ops/list/src/lib.rs @@ -1,12 +1,15 @@ +use linode::util::JobNodeConfig; use proto::backend::{self, pkg::*}; use rivet_operation::prelude::*; -use util_cluster::JobNodeConfig; #[operation(name = "tier-list")] async fn handle(ctx: OperationContext) -> GlobalResult { - let datacenters_res = op!([ctx] cluster_datacenter_get { - datacenter_ids: ctx.region_ids.clone(), - }) + let datacenters_res = chirp_workflow::compat::op( + &ctx, + cluster::ops::datacenter::get::Input { + datacenter_ids: ctx.region_ids.iter().map(|id| id.as_uuid()).collect(), + }, + ) .await?; let hardware = datacenters_res @@ -16,7 +19,7 @@ async fn handle(ctx: OperationContext) -> GlobalResult) -> GlobalResult>>()?; - let instance_types_res = op!([ctx] linode_instance_type_get { - hardware_ids: hardware - .iter() - .map(|(_, hardware)| hardware.clone()) - .collect::>(), - }) + let instance_types_res = chirp_workflow::compat::op( + &ctx, + linode::ops::instance_type_get::Input { + hardware_ids: hardware + .iter() + .map(|(_, hardware)| hardware.clone()) + .collect::>(), + }, + ) .await?; let regions = hardware @@ -49,17 +55,16 @@ async fn handle(ctx: OperationContext) -> GlobalResult