diff --git a/CHANGELOG.md b/CHANGELOG.md index 369cdb5..c501406 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - Implemented basic authentication for configurable endpoint paths (#73) +- Added integrations guide with example `testlight` integration ## [1.2.0] - 2025-10-14 diff --git a/Cargo.lock b/Cargo.lock index 0844243..f5578c3 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -2355,6 +2355,7 @@ checksum = "df8b2b54733674ad286d16267dcfc7a71ed5c776e4ac7aa3c3e2561f7c637bf2" name = "trusted-server-common" version = "0.1.0" dependencies = [ + "async-trait", "base64 0.22.1", "brotli", "bytes", @@ -2376,6 +2377,7 @@ dependencies = [ "log", "log-fastly", "lol_html", + "once_cell", "pin-project-lite", "rand", "regex", diff --git a/Cargo.toml b/Cargo.toml index 44b723c..8d5db5b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -16,6 +16,7 @@ default-members = [ debug = 1 [workspace.dependencies] +async-trait = "0.1" base64 = "0.22" brotli = "8.0" bytes = "1.10" @@ -34,13 +35,14 @@ handlebars = "6.3.2" hex = "0.4.3" hmac = "0.12.1" http = "1.3.1" +jose-jwk = "0.1.2" log = "0.4.28" log-fastly = "0.11.9" lol_html = "2.7.0" +once_cell = "1.19" pin-project-lite = "0.2" -regex = "1.12.2" -jose-jwk = "0.1.2" rand = "0.8" +regex = "1.12.2" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0.145" sha2 = "0.10.9" diff --git a/FAQ_POC.md b/FAQ_POC.md index 39e4f2f..b79c1bf 100644 --- a/FAQ_POC.md +++ b/FAQ_POC.md @@ -25,7 +25,7 @@ NOT all the capabilities. Tech Lab will build and support core services to enabl **Does the Trusteed Server preclude using third party tags?** No, it does not. You should be able to begin migrating certain modules and parts of your content and experience as you go, without a forklift upgrade. We plan to support server side tagging capabilities to enable third party support. -**Why are you only using two vendors in the POC?** +**Why are you only using two partners in the POC?** Fastly and Equativ volunteered time and resources to us and they fit the technical needs and requirements for Trusted Server. For the sake of getting to market ASAP, we chose to double down on these two partners. We do not play favorites or have any financial incentive with these two companies and will begin implementing on other partners in the near future. Any ad exchange supporting prebid server requests should already find support. We will prioritize modules for other edge cloud providers based on industry priorities **How will this project be managed?** @@ -39,4 +39,3 @@ Yes. As long as your managed service provider can separate the edge from the CMS **How will this comply with Privacy regulations?** The trusted server will have modules to support Consent Management Providers (CMP) and send the GPP or TCF string as required in the ad request. - diff --git a/README.md b/README.md index d39abc1..49f1dc8 100644 --- a/README.md +++ b/README.md @@ -1,29 +1,30 @@ -# Trusted Server +# Trusted Server -:information_source: Trusted Server is an open-source, cloud based orchestration framework and runtime for publishers. It moves code execution and operations that traditionally occurs in browsers (via 3rd party JS) to secure, zero-cold-start [WASM](https://webassembly.org) binaries running in [WASI](https://github.com/WebAssembly/WASI) supported environments. It importantly gives publishers benefits such as: dramatically increasing control over how and who they share their data with (while maintaining user-privacy compliance), increasing revenue from inventory inside cookie restricted or non-JS environments, ability to serve all assets under 1st party context, and provides secure cryptographic functions to ensure trust across the programmatic ad ecosystem. +:information_source: Trusted Server is an open-source, cloud based orchestration framework and runtime for publishers. It moves code execution and operations that traditionally occurs in browsers (via 3rd party JS) to secure, zero-cold-start [WASM](https://webassembly.org) binaries running in [WASI](https://github.com/WebAssembly/WASI) supported environments. It importantly gives publishers benefits such as: dramatically increasing control over how and who they share their data with (while maintaining user-privacy compliance), increasing revenue from inventory inside cookie restricted or non-JS environments, ability to serve all assets under 1st party context, and provides secure cryptographic functions to ensure trust across the programmatic ad ecosystem. Trusted Server is the new execution layer for the open-web, returning control of 1st party data, security, and overall user-experience back to publishers. At this time, Trusted Server is designed to work with Fastly Compute. Follow these steps to configure Fastly Compute and deploy it. ## Getting Started: Edge-Cloud Support on Fastly + - Create account at Fastly if you don’t have one - manage.fastly.com -- Log in to the Fastly control panel. - - Go to Account > API tokens > Personal tokens. - - Click Create token - - Name the Token - - Choose User Token - - Choose Global API Access - - Choose what makes sense for your Org in terms of Service Access - - Copy key to a secure location because you will not be able to see it again - -- Create new Compute Service - - Click Compute and Create Service - - Click “Create Empty Service” (below main options) - - Add your domain of the website you’ll be testing or using and click update - - Click on “Origins” section and add your ad-server / ssp partner information as hostnames (note after you save this information you can select port numbers and TLS on/off) - - IMPORTANT: when you enter the FQDN or IP ADDR information and click Add you need to enter a “Name” in the first field that will be referenced in your code so something like “my_ad_partner_1” - - +- Log in to the Fastly control panel. + - Go to Account > API tokens > Personal tokens. + - Click Create token + - Name the Token + - Choose User Token + - Choose Global API Access + - Choose what makes sense for your Org in terms of Service Access + - Copy key to a secure location because you will not be able to see it again + +- Create new Compute Service + - Click Compute and Create Service + - Click “Create Empty Service” (below main options) + - Add your domain of the website you’ll be testing or using and click update + - Click on “Origins” section and add your ad-server / SSP integration information as hostnames (note after you save this information you can select port numbers and TLS on/off) + - IMPORTANT: when you enter the FQDN or IP ADDR information and click Add you need to enter a “Name” in the first field that will be referenced in your code so something like “my_ad_integration_1” + - :warning: With a dev account, Fastly gives you a test domain by default, but you’re also able to create a CNAME to your own domain when you’re ready, along with 2 free TLS certs (non-wildcard). Note that Fastly Compute ONLY accepts client traffic via TLS, though origins and backends can be non-TLS. @@ -38,14 +39,17 @@ At this time, Trusted Server is designed to work with Fastly Compute. Follow the ```sh /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" ``` + ### Fastly CLI -#### Install Fastly CLI +#### Install Fastly CLI + ```sh brew install fastly/tap/fastly ``` -#### Verify Installation and Version +#### Verify Installation and Version + ```sh fastly version ``` @@ -53,13 +57,15 @@ fastly version :warning: fastly cli version should be at least v12.1.0 #### Create profile and follow interactive prompt for pasting your API Token created earlier: -```sh + +```sh fastly profile create ``` ### Rust #### Install Rust with asdf (our preference) + ```sh brew install asdf asdf plugin add rust @@ -70,6 +76,7 @@ asdf reshim ### NodeJS #### Install NodeJS with asdf + ```sh brew install asdf asdf plugin add nodejs @@ -79,34 +86,39 @@ asdf reshim #### Fix path for Bash -Edit ~/.bash_profile to add path for asdf shims: +Edit ~/.bash_profile to add path for asdf shims: + ```sh export PATH="${ASDF_DATA_DIR:-$HOME/.asdf}/shims:$PATH" ``` #### Fix path for ZSH -Edit ~/.zshrc to add path for asdf shims: +Edit ~/.zshrc to add path for asdf shims: + ```sh export PATH="${ASDF_DATA_DIR:-$HOME/.asdf}/shims:$PATH" ``` #### Other shells -See https://asdf-vm.com/guide/getting-started.html#_2-configure-asdf +See https://asdf-vm.com/guide/getting-started.html#_2-configure-asdf -### Clone Trusted Server and Configure Build +### Clone Trusted Server and Configure Build #### Clone Project (assumes you have 'git' installed on your system) + ```sh git clone git@github.com:IABTechLab/trusted-server.git ``` ### Configure + #### Edit configuration files + :information_source: Note that you'll have to edit the following files for your setup: -- fastly.toml (service ID, author, description, Config/Secret Store IDs for request signing) +- fastly.toml (service ID, author, description, Config/Secret Store IDs for request signing) - trusted-server.toml (KV store ID names - optional, request signing configuration) ### Build @@ -124,11 +136,13 @@ fastly compute publish ## Devleopment #### Install viceroy for running tests + ```sh cargo install viceroy ``` #### Run Fastly server locally + - Review configuration for [local_server](fastly.toml#L16) - Review env variables overrides in [.env.dev](.env.dev) @@ -141,6 +155,7 @@ fastly -i compute serve ``` #### Tests + ```sh cargo test ``` @@ -148,6 +163,7 @@ cargo test :warning: if test fails `viceroy` will not display line number of the failed test. Rerun it with `cargo test_details`. #### Additional Rust Commands + - `cargo fmt`: Ensure uniform code formatting - `cargo clippy`: Ensure idiomatic code - `cargo check`: Ensure compilation succeeds on Linux, MacOS, Windows and WebAssembly @@ -166,6 +182,7 @@ Request signing requires Fastly Config Store and Secret Store for key management - Secret Store: `signing_keys` - stores private signing keys 2. **Configure in trusted-server.toml**: + ```toml [request_signing] enabled = true # Set to true to enable request signing @@ -187,7 +204,6 @@ Once configured, the following endpoints are available: - **`POST /admin/keys/rotate`**: Generates and activates a new signing key - Optional body: `{"kid": "custom-key-id"}` (auto-generates date-based ID if omitted) - Response includes new key ID, previous key ID, and active keys list - - **`POST /admin/keys/deactivate`**: Deactivates or deletes a key - Request body: `{"kid": "key-to-deactivate", "delete": false}` - Set `delete: true` to permanently remove the key (also deactivates it) @@ -196,9 +212,9 @@ Once configured, the following endpoints are available: ## First-Party Endpoints - - `/first-party/ad` (GET): returns HTML for a single slot (`slot`, `w`, `h` query params). The server inspects returned creative HTML and rewrites: - - All absolute images and iframes to `/first-party/proxy?tsurl=&&tstoken=` (1×1 pixels are detected server‑side heuristically for logging). The `tstoken` is derived from encrypting the full target URL and hashing it. - - `/third-party/ad` (POST): accepts tsjs ad units and proxies to Prebid Server. +- `/first-party/ad` (GET): returns HTML for a single slot (`slot`, `w`, `h` query params). The server inspects returned creative HTML and rewrites: +- All absolute images and iframes to `/first-party/proxy?tsurl=&&tstoken=` (1×1 pixels are detected server‑side heuristically for logging). The `tstoken` is derived from encrypting the full target URL and hashing it. +- `/third-party/ad` (POST): accepts tsjs ad units and proxies to Prebid Server. - `/first-party/proxy` (GET): unified proxy for resources referenced by creatives. - Query params: - `tsurl`: Target URL without query (base URL) — required @@ -223,7 +239,12 @@ Once configured, the following endpoints are available: - Publisher origin proxy (`handle_publisher_request`): retrieves/generates the synthetic ID, stamps the response with `X-Synthetic-*` headers, and sets the `synthetic_id` cookie (Secure, SameSite=Lax) when absent so subsequent creative and click proxies can propagate the identifier. - Notes - - Rewriting uses `lol_html`. Only absolute and protocol‑relative URLs are rewritten; relative URLs are left unchanged. - - For the proxy endpoint, the base URL is carried in `tsurl`, the original query parameters are preserved individually, and `tstoken` authenticates the reconstructed full URL. - - Synthetic identifiers are generated by `crates/common/src/synthetic.rs` and are surfaced in three places: publisher responses (headers + cookie), creative proxy target URLs (`synthetic_id` query param), and click redirect URLs. This ensures downstream partners can correlate impressions and clicks without direct third-party cookies. +Notes + +- Rewriting uses `lol_html`. Only absolute and protocol‑relative URLs are rewritten; relative URLs are left unchanged. +- For the proxy endpoint, the base URL is carried in `tsurl`, the original query parameters are preserved individually, and `tstoken` authenticates the reconstructed full URL. +- Synthetic identifiers are generated by `crates/common/src/synthetic.rs` and are surfaced in three places: publisher responses (headers + cookie), creative proxy target URLs (`synthetic_id` query param), and click redirect URLs. This ensures downstream integrations can correlate impressions and clicks without direct third-party cookies. + +## Integration Modules + +- See [`docs/integration_guide.md`](docs/integration_guide.md) for the full integration module guide, covering configuration, proxy routing, HTML shim hooks, and the `testlight` example implementation. diff --git a/crates/common/Cargo.toml b/crates/common/Cargo.toml index cdf21e3..33b7f97 100644 --- a/crates/common/Cargo.toml +++ b/crates/common/Cargo.toml @@ -14,6 +14,7 @@ brotli = { workspace = true } bytes = { workspace = true } chacha20poly1305 = { workspace = true } chrono = { workspace = true } +async-trait = { workspace = true } config = { workspace = true } cookie = { workspace = true } derive_more = { workspace = true } @@ -42,6 +43,7 @@ urlencoding = { workspace = true } uuid = { workspace = true } validator = { workspace = true } ed25519-dalek = { workspace = true } +once_cell = { workspace = true } [build-dependencies] config = { workspace = true } diff --git a/crates/common/src/creative.rs b/crates/common/src/creative.rs index c6c57f1..3953b2e 100644 --- a/crates/common/src/creative.rs +++ b/crates/common/src/creative.rs @@ -300,12 +300,12 @@ pub fn rewrite_creative_html(markup: &str, settings: &Settings) -> String { let mut rewriter = HtmlRewriter::new( HtmlSettings { element_content_handlers: vec![ - // Inject tsjs-creative at the top of body once + // Inject unified tsjs bundle at the top of body once element!("body", { let injected = injected_ts_creative.clone(); move |el| { if !injected.get() { - let script_tag = tsjs::creative_script_tag(); + let script_tag = tsjs::unified_script_tag(); el.prepend(&script_tag, ContentType::Html); injected.set(true); } @@ -490,20 +490,20 @@ mod tests { let html = r#"

hello

"#; let out = rewrite_creative_html(html, &settings); assert!( - out.contains("/static/tsjs=tsjs-creative.min.js"), - "expected tsjs-creative injection: {}", + out.contains("/static/tsjs=tsjs-unified.min.js"), + "expected unified tsjs injection: {}", out ); // Inject only once - assert_eq!(out.matches("/static/tsjs=tsjs-creative.min.js").count(), 1); + assert_eq!(out.matches("/static/tsjs=tsjs-unified.min.js").count(), 1); } #[test] - fn injects_tsjs_creative_once_with_multiple_bodies() { + fn injects_tsjs_unified_once_with_multiple_bodies() { let settings = crate::test_support::tests::create_test_settings(); let html = r#"onetwo"#; let out = rewrite_creative_html(html, &settings); - assert_eq!(out.matches("/static/tsjs=tsjs-creative.min.js").count(), 1); + assert_eq!(out.matches("/static/tsjs=tsjs-unified.min.js").count(), 1); } #[test] diff --git a/crates/common/src/error.rs b/crates/common/src/error.rs index 671f422..b57bc5d 100644 --- a/crates/common/src/error.rs +++ b/crates/common/src/error.rs @@ -50,6 +50,13 @@ pub enum TrustedServerError { #[display("Prebid error: {message}")] Prebid { message: String }, + /// Integration module error. + #[display("Integration error ({integration}): {message}")] + Integration { + integration: String, + message: String, + }, + /// Proxy error. #[display("Proxy error: {message}")] Proxy { message: String }, @@ -91,6 +98,7 @@ impl IntoHttpResponse for TrustedServerError { Self::InvalidUtf8 { .. } => StatusCode::BAD_REQUEST, Self::KvStore { .. } => StatusCode::SERVICE_UNAVAILABLE, Self::Prebid { .. } => StatusCode::BAD_GATEWAY, + Self::Integration { .. } => StatusCode::BAD_GATEWAY, Self::Proxy { .. } => StatusCode::BAD_GATEWAY, Self::SyntheticId { .. } => StatusCode::INTERNAL_SERVER_ERROR, Self::Template { .. } => StatusCode::INTERNAL_SERVER_ERROR, diff --git a/crates/common/src/html_processor.rs b/crates/common/src/html_processor.rs index 5325a1e..d59ee8b 100644 --- a/crates/common/src/html_processor.rs +++ b/crates/common/src/html_processor.rs @@ -7,6 +7,10 @@ use std::rc::Rc; use lol_html::{element, html_content::ContentType, text, Settings as RewriterSettings}; use regex::Regex; +use crate::integrations::{ + AttributeRewriteOutcome, IntegrationAttributeContext, IntegrationRegistry, + IntegrationScriptContext, ScriptRewriteAction, +}; use crate::settings::Settings; use crate::streaming_processor::{HtmlRewriterAdapter, StreamProcessor}; use crate::tsjs; @@ -18,6 +22,7 @@ pub struct HtmlProcessorConfig { pub request_host: String, pub request_scheme: String, pub enable_prebid: bool, + pub integrations: IntegrationRegistry, pub nextjs_enabled: bool, pub nextjs_attributes: Vec, } @@ -26,6 +31,7 @@ impl HtmlProcessorConfig { /// Create from settings and request parameters pub fn from_settings( settings: &Settings, + integrations: &IntegrationRegistry, origin_host: &str, request_host: &str, request_scheme: &str, @@ -35,6 +41,7 @@ impl HtmlProcessorConfig { request_host: request_host.to_string(), request_scheme: request_scheme.to_string(), enable_prebid: settings.prebid.auto_configure, + integrations: integrations.clone(), nextjs_enabled: settings.publisher.nextjs.enabled, nextjs_attributes: settings.publisher.nextjs.rewrite_attributes.clone(), } @@ -114,6 +121,8 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso let nextjs_attributes = Rc::new(config.nextjs_attributes.clone()); let injected_tsjs = Rc::new(Cell::new(false)); + let integration_registry = config.integrations.clone(); + let script_rewriters = integration_registry.script_rewriters(); fn is_prebid_script_url(url: &str) -> bool { let lower = url.to_ascii_lowercase(); @@ -126,12 +135,12 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso } let mut element_content_handlers = vec![ - // Inject tsjs once at the start of + // Inject unified tsjs bundle once at the start of element!("head", { let injected_tsjs = injected_tsjs.clone(); move |el| { if !injected_tsjs.get() { - let loader = tsjs::core_script_tag(); + let loader = tsjs::unified_script_tag(); el.prepend(&loader, ContentType::Html); injected_tsjs.set(true); } @@ -142,20 +151,45 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso element!("[href]", { let patterns = patterns.clone(); let rewrite_prebid = config.enable_prebid; + let integrations = integration_registry.clone(); move |el| { - if let Some(href) = el.get_attribute("href") { - // If Prebid auto-config is enabled and this looks like a Prebid script href, rewrite to our extension + if let Some(mut href) = el.get_attribute("href") { + let original_href = href.clone(); if rewrite_prebid && is_prebid_script_url(&href) { - let ext_src = tsjs::ext_script_src(); - el.set_attribute("href", &ext_src)?; + el.remove(); + return Ok(()); } else { let new_href = href .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()); if new_href != href { - el.set_attribute("href", &new_href)?; + href = new_href; } } + + match integrations.rewrite_attribute( + "href", + &href, + &IntegrationAttributeContext { + attribute_name: "href", + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }, + ) { + AttributeRewriteOutcome::Unchanged => {} + AttributeRewriteOutcome::Replaced(integration_href) => { + href = integration_href; + } + AttributeRewriteOutcome::RemoveElement => { + el.remove(); + return Ok(()); + } + } + + if href != original_href { + el.set_attribute("href", &href)?; + } } Ok(()) } @@ -164,19 +198,45 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso element!("[src]", { let patterns = patterns.clone(); let rewrite_prebid = config.enable_prebid; + let integrations = integration_registry.clone(); move |el| { - if let Some(src) = el.get_attribute("src") { + if let Some(mut src) = el.get_attribute("src") { + let original_src = src.clone(); if rewrite_prebid && is_prebid_script_url(&src) { - let ext_src = tsjs::ext_script_src(); - el.set_attribute("src", &ext_src)?; + el.remove(); + return Ok(()); } else { let new_src = src .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()); if new_src != src { - el.set_attribute("src", &new_src)?; + src = new_src; + } + } + + match integrations.rewrite_attribute( + "src", + &src, + &IntegrationAttributeContext { + attribute_name: "src", + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }, + ) { + AttributeRewriteOutcome::Unchanged => {} + AttributeRewriteOutcome::Replaced(integration_src) => { + src = integration_src; + } + AttributeRewriteOutcome::RemoveElement => { + el.remove(); + return Ok(()); } } + + if src != original_src { + el.set_attribute("src", &src)?; + } } Ok(()) } @@ -184,13 +244,39 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in action attributes element!("[action]", { let patterns = patterns.clone(); + let integrations = integration_registry.clone(); move |el| { - if let Some(action) = el.get_attribute("action") { + if let Some(mut action) = el.get_attribute("action") { + let original_action = action.clone(); let new_action = action .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()); if new_action != action { - el.set_attribute("action", &new_action)?; + action = new_action; + } + + match integrations.rewrite_attribute( + "action", + &action, + &IntegrationAttributeContext { + attribute_name: "action", + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }, + ) { + AttributeRewriteOutcome::Unchanged => {} + AttributeRewriteOutcome::Replaced(integration_action) => { + action = integration_action; + } + AttributeRewriteOutcome::RemoveElement => { + el.remove(); + return Ok(()); + } + } + + if action != original_action { + el.set_attribute("action", &action)?; } } Ok(()) @@ -199,8 +285,10 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in srcset attributes (for responsive images) element!("[srcset]", { let patterns = patterns.clone(); + let integrations = integration_registry.clone(); move |el| { - if let Some(srcset) = el.get_attribute("srcset") { + if let Some(mut srcset) = el.get_attribute("srcset") { + let original_srcset = srcset.clone(); let new_srcset = srcset .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()) @@ -209,9 +297,32 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso &patterns.protocol_relative_replacement(), ) .replace(&patterns.origin_host, &patterns.request_host); - if new_srcset != srcset { - el.set_attribute("srcset", &new_srcset)?; + srcset = new_srcset; + } + + match integrations.rewrite_attribute( + "srcset", + &srcset, + &IntegrationAttributeContext { + attribute_name: "srcset", + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }, + ) { + AttributeRewriteOutcome::Unchanged => {} + AttributeRewriteOutcome::Replaced(integration_srcset) => { + srcset = integration_srcset; + } + AttributeRewriteOutcome::RemoveElement => { + el.remove(); + return Ok(()); + } + } + + if srcset != original_srcset { + el.set_attribute("srcset", &srcset)?; } } Ok(()) @@ -220,8 +331,10 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso // Replace URLs in imagesrcset attributes (for link preload) element!("[imagesrcset]", { let patterns = patterns.clone(); + let integrations = integration_registry.clone(); move |el| { - if let Some(imagesrcset) = el.get_attribute("imagesrcset") { + if let Some(mut imagesrcset) = el.get_attribute("imagesrcset") { + let original_imagesrcset = imagesrcset.clone(); let new_imagesrcset = imagesrcset .replace(&patterns.https_origin(), &patterns.replacement_url()) .replace(&patterns.http_origin(), &patterns.replacement_url()) @@ -230,7 +343,31 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso &patterns.protocol_relative_replacement(), ); if new_imagesrcset != imagesrcset { - el.set_attribute("imagesrcset", &new_imagesrcset)?; + imagesrcset = new_imagesrcset; + } + + match integrations.rewrite_attribute( + "imagesrcset", + &imagesrcset, + &IntegrationAttributeContext { + attribute_name: "imagesrcset", + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }, + ) { + AttributeRewriteOutcome::Unchanged => {} + AttributeRewriteOutcome::Replaced(integration_imagesrcset) => { + imagesrcset = integration_imagesrcset; + } + AttributeRewriteOutcome::RemoveElement => { + el.remove(); + return Ok(()); + } + } + + if imagesrcset != original_imagesrcset { + el.set_attribute("imagesrcset", &imagesrcset)?; } } Ok(()) @@ -238,6 +375,34 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso }), ]; + for script_rewriter in script_rewriters { + let selector = script_rewriter.selector(); + let rewriter = script_rewriter.clone(); + let patterns = patterns.clone(); + element_content_handlers.push(text!(selector, { + let rewriter = rewriter.clone(); + let patterns = patterns.clone(); + move |text| { + let ctx = IntegrationScriptContext { + selector, + request_host: &patterns.request_host, + request_scheme: &patterns.request_scheme, + origin_host: &patterns.origin_host, + }; + match rewriter.rewrite(text.as_str(), &ctx) { + ScriptRewriteAction::Keep => {} + ScriptRewriteAction::Replace(rewritten) => { + text.replace(&rewritten, ContentType::Text); + } + ScriptRewriteAction::RemoveNode => { + text.remove(); + } + } + Ok(()) + } + })); + } + if config.nextjs_enabled && !nextjs_attributes.is_empty() { element_content_handlers.push(text!("script#__NEXT_DATA__", { let patterns = patterns.clone(); @@ -269,28 +434,6 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso let rewriter_settings = RewriterSettings { element_content_handlers, - - // TODO: Consider adding text content replacement if needed with settings - // // Replace URLs in text content - // document_content_handlers: vec![lol_html::doc_text!({ - // move |text| { - // let content = text.as_str(); - - // // Apply URL replacements - // let mut new_content = content.to_string(); - // for replacement in replacer.replacements.iter() { - // if new_content.contains(&replacement.find) { - // new_content = new_content.replace(&replacement.find, &replacement.replace_with); - // } - // } - - // if new_content != content { - // text.replace(&new_content, lol_html::html_content::ContentType::Text); - // } - - // Ok(()) - // } - // })], ..RewriterSettings::default() }; @@ -300,8 +443,10 @@ pub fn create_html_processor(config: HtmlProcessorConfig) -> impl StreamProcesso #[cfg(test)] mod tests { use super::*; + use crate::integrations::{AttributeRewriteAction, IntegrationAttributeRewriter}; use crate::streaming_processor::{Compression, PipelineConfig, StreamingPipeline}; use std::io::Cursor; + use std::sync::Arc; fn create_test_config() -> HtmlProcessorConfig { HtmlProcessorConfig { @@ -309,20 +454,81 @@ mod tests { request_host: "test.example.com".to_string(), request_scheme: "https".to_string(), enable_prebid: false, + integrations: IntegrationRegistry::default(), nextjs_enabled: false, nextjs_attributes: vec!["href".to_string(), "link".to_string(), "url".to_string()], } } #[test] - fn test_injects_tsjs_script_and_rewrites_prebid_refs() { + fn integration_attribute_rewriter_can_remove_elements() { + struct RemovingLinkRewriter; + + impl IntegrationAttributeRewriter for RemovingLinkRewriter { + fn integration_id(&self) -> &'static str { + "removing" + } + + fn handles_attribute(&self, attribute: &str) -> bool { + attribute == "href" + } + + fn rewrite( + &self, + _attr_name: &str, + attr_value: &str, + _ctx: &IntegrationAttributeContext<'_>, + ) -> AttributeRewriteAction { + if attr_value.contains("remove-me") { + AttributeRewriteAction::remove_element() + } else { + AttributeRewriteAction::keep() + } + } + } + + let html = r#" + remove + keep + "#; + + let mut config = create_test_config(); + config.integrations = + IntegrationRegistry::from_rewriters(vec![Arc::new(RemovingLinkRewriter)], Vec::new()); + + let processor = create_html_processor(config); + let pipeline_config = PipelineConfig { + input_compression: Compression::None, + output_compression: Compression::None, + chunk_size: 8192, + }; + let mut pipeline = StreamingPipeline::new(pipeline_config, processor); + + let mut output = Vec::new(); + pipeline + .process(Cursor::new(html.as_bytes()), &mut output) + .unwrap(); + let processed = String::from_utf8(output).unwrap(); + + assert!( + processed.contains("keep-me"), + "Expected keep link to remain" + ); + assert!( + !processed.contains("remove-me"), + "Removing rewriter should drop matching elements" + ); + } + + #[test] + fn test_injects_unified_bundle_and_removes_prebid_refs() { let html = r#" "#; let mut config = create_test_config(); - config.enable_prebid = true; // enable rewriting of Prebid URLs + config.enable_prebid = true; // enable removal of Prebid URLs let processor = create_html_processor(config); let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -335,19 +541,20 @@ mod tests { let result = pipeline.process(Cursor::new(html.as_bytes()), &mut output); assert!(result.is_ok()); let processed = String::from_utf8_lossy(&output); - assert!(processed.contains("/static/tsjs=tsjs-core.min.js")); - // Prebid references are rewritten to our extension when auto-configure is on - assert!(processed.contains("/static/tsjs=tsjs-ext.min.js")); + assert!(processed.contains("/static/tsjs=tsjs-unified.min.js")); + // Prebid script references should be removed when auto-configure is on + assert!(!processed.contains("prebid.min.js")); + assert!(!processed.contains("cdn.prebid.org/prebid.js")); } #[test] - fn test_injects_tsjs_script_and_rewrites_prebid_with_query_string() { + fn test_injects_unified_bundle_and_removes_prebid_with_query_string() { let html = r#" "#; let mut config = create_test_config(); - config.enable_prebid = true; // enable rewriting of Prebid URLs + config.enable_prebid = true; // enable removal of Prebid URLs let processor = create_html_processor(config); let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -360,19 +567,21 @@ mod tests { let result = pipeline.process(Cursor::new(html.as_bytes()), &mut output); assert!(result.is_ok()); let processed = String::from_utf8_lossy(&output); - assert!(processed.contains("/static/tsjs=tsjs-core.min.js")); - assert!(processed.contains("/static/tsjs=tsjs-ext.min.js")); + // Should inject unified bundle + assert!(processed.contains("/static/tsjs=tsjs-unified.min.js")); + // Prebid script should be removed + assert!(!processed.contains("prebidjs.min.js")); } #[test] - fn test_always_injects_tsjs_script() { + fn test_always_injects_unified_bundle() { let html = r#" "#; let mut config = create_test_config(); - config.enable_prebid = false; // No longer affects tsjs injection + config.enable_prebid = false; // When disabled, don't remove Prebid scripts let processor = create_html_processor(config); let pipeline_config = PipelineConfig { input_compression: Compression::None, @@ -385,10 +594,11 @@ mod tests { let result = pipeline.process(Cursor::new(html.as_bytes()), &mut output); assert!(result.is_ok()); let processed = String::from_utf8_lossy(&output); - // When auto-configure is disabled, do not rewrite Prebid references + // When auto-configure is disabled, do not remove Prebid references assert!(processed.contains("/js/prebid.min.js")); assert!(processed.contains("cdn.prebid.org/prebid.js")); - assert!(processed.contains("/static/tsjs=tsjs-core.min.js")); + // But still inject unified bundle + assert!(processed.contains("/static/tsjs=tsjs-unified.min.js")); } #[test] @@ -558,8 +768,10 @@ mod tests { use crate::test_support::tests::create_test_settings; let settings = create_test_settings(); + let integrations = IntegrationRegistry::default(); let config = HtmlProcessorConfig::from_settings( &settings, + &integrations, "origin.test-publisher.com", "proxy.example.com", "https", diff --git a/crates/common/src/integrations/mod.rs b/crates/common/src/integrations/mod.rs new file mode 100644 index 0000000..e4577c4 --- /dev/null +++ b/crates/common/src/integrations/mod.rs @@ -0,0 +1,19 @@ +//! Integration module registry and sample implementations. + +use crate::settings::Settings; + +mod registry; +pub mod testlight; + +pub use registry::{ + AttributeRewriteAction, AttributeRewriteOutcome, IntegrationAttributeContext, + IntegrationAttributeRewriter, IntegrationEndpoint, IntegrationMetadata, IntegrationProxy, + IntegrationRegistration, IntegrationRegistrationBuilder, IntegrationRegistry, + IntegrationScriptContext, IntegrationScriptRewriter, ScriptRewriteAction, +}; + +type IntegrationBuilder = fn(&Settings) -> Option; + +pub(crate) fn builders() -> &'static [IntegrationBuilder] { + &[testlight::register] +} diff --git a/crates/common/src/integrations/registry.rs b/crates/common/src/integrations/registry.rs new file mode 100644 index 0000000..b13e610 --- /dev/null +++ b/crates/common/src/integrations/registry.rs @@ -0,0 +1,402 @@ +use std::collections::{BTreeMap, HashMap}; +use std::sync::Arc; + +use async_trait::async_trait; +use error_stack::Report; +use fastly::http::Method; +use fastly::{Request, Response}; + +use crate::error::TrustedServerError; +use crate::settings::Settings; + +/// Action returned by attribute rewriters to describe how the runtime should mutate the element. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum AttributeRewriteAction { + /// Leave the attribute and element untouched. + Keep, + /// Replace the attribute value with the provided string. + Replace(String), + /// Remove the entire element from the HTML stream. + RemoveElement, +} + +impl AttributeRewriteAction { + #[must_use] + pub fn keep() -> Self { + Self::Keep + } + + #[must_use] + pub fn replace(value: impl Into) -> Self { + Self::Replace(value.into()) + } + + #[must_use] + pub fn remove_element() -> Self { + Self::RemoveElement + } +} + +/// Outcome returned by the registry after running every matching attribute rewriter. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum AttributeRewriteOutcome { + Unchanged, + Replaced(String), + RemoveElement, +} + +/// Action returned by inline script rewriters to describe how to mutate the node. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum ScriptRewriteAction { + Keep, + Replace(String), + RemoveNode, +} + +impl ScriptRewriteAction { + #[must_use] + pub fn keep() -> Self { + Self::Keep + } + + #[must_use] + pub fn replace(value: impl Into) -> Self { + Self::Replace(value.into()) + } + + #[must_use] + pub fn remove_node() -> Self { + Self::RemoveNode + } +} + +/// Context provided to integration HTML attribute rewriters. +#[derive(Debug)] +pub struct IntegrationAttributeContext<'a> { + pub attribute_name: &'a str, + pub request_host: &'a str, + pub request_scheme: &'a str, + pub origin_host: &'a str, +} + +/// Context passed to script/text rewriters for inline HTML handling. +#[derive(Debug)] +pub struct IntegrationScriptContext<'a> { + pub selector: &'a str, + pub request_host: &'a str, + pub request_scheme: &'a str, + pub origin_host: &'a str, +} + +/// Describes an HTTP endpoint exposed by an integration. +#[derive(Clone, Debug)] +pub struct IntegrationEndpoint { + pub method: Method, + pub path: &'static str, +} + +impl IntegrationEndpoint { + #[must_use] + pub fn new(method: Method, path: &'static str) -> Self { + Self { method, path } + } + + #[must_use] + pub fn get(path: &'static str) -> Self { + Self { + method: Method::GET, + path, + } + } + + #[must_use] + pub fn post(path: &'static str) -> Self { + Self { + method: Method::POST, + path, + } + } +} + +/// Trait implemented by integration proxies that expose HTTP endpoints. +#[async_trait(?Send)] +pub trait IntegrationProxy: Send + Sync { + /// Routes handled by this integration (e.g. `/integrations/example/auction`). + fn routes(&self) -> Vec; + + /// Handle the proxied request. + async fn handle( + &self, + settings: &Settings, + req: Request, + ) -> Result>; +} + +/// Trait for integration-provided HTML attribute rewrite hooks. +pub trait IntegrationAttributeRewriter: Send + Sync { + /// Identifier for logging/diagnostics. + fn integration_id(&self) -> &'static str; + /// Return true when this rewriter wants to inspect a given attribute. + fn handles_attribute(&self, attribute: &str) -> bool; + /// Attempt to rewrite the attribute value. Return `AttributeRewriteAction::Replace` + /// to update the attribute, `Keep` to leave it untouched, or `RemoveElement` to drop the node. + fn rewrite( + &self, + attr_name: &str, + attr_value: &str, + ctx: &IntegrationAttributeContext<'_>, + ) -> AttributeRewriteAction; +} + +/// Trait for integration-provided inline script/text rewrite hooks. +pub trait IntegrationScriptRewriter: Send + Sync { + /// Identifier for logging/diagnostics. + fn integration_id(&self) -> &'static str; + /// CSS selector (e.g. `script#__NEXT_DATA__`) that should trigger this rewriter. + fn selector(&self) -> &'static str; + /// Attempt to rewrite the inline text content for the selector. + fn rewrite(&self, content: &str, ctx: &IntegrationScriptContext<'_>) -> ScriptRewriteAction; +} + +/// Registration payload returned by integration builders. +pub struct IntegrationRegistration { + pub integration_id: &'static str, + pub proxies: Vec>, + pub attribute_rewriters: Vec>, + pub script_rewriters: Vec>, +} + +impl IntegrationRegistration { + #[must_use] + pub fn builder(integration_id: &'static str) -> IntegrationRegistrationBuilder { + IntegrationRegistrationBuilder::new(integration_id) + } +} + +pub struct IntegrationRegistrationBuilder { + registration: IntegrationRegistration, +} + +impl IntegrationRegistrationBuilder { + fn new(integration_id: &'static str) -> Self { + Self { + registration: IntegrationRegistration { + integration_id, + proxies: Vec::new(), + attribute_rewriters: Vec::new(), + script_rewriters: Vec::new(), + }, + } + } + + #[must_use] + pub fn with_proxy(mut self, proxy: Arc) -> Self { + self.registration.proxies.push(proxy); + self + } + + #[must_use] + pub fn with_attribute_rewriter( + mut self, + rewriter: Arc, + ) -> Self { + self.registration.attribute_rewriters.push(rewriter); + self + } + + #[must_use] + pub fn with_script_rewriter(mut self, rewriter: Arc) -> Self { + self.registration.script_rewriters.push(rewriter); + self + } + + #[must_use] + pub fn build(self) -> IntegrationRegistration { + self.registration + } +} + +type RouteKey = (Method, String); +type RouteValue = (Arc, &'static str); + +#[derive(Default)] +struct IntegrationRegistryInner { + route_map: HashMap, + routes: Vec<(IntegrationEndpoint, &'static str)>, + html_rewriters: Vec>, + script_rewriters: Vec>, +} + +/// Summary of registered integration capabilities. +#[derive(Debug, Clone)] +pub struct IntegrationMetadata { + pub id: &'static str, + pub routes: Vec, + pub attribute_rewriters: usize, + pub script_selectors: Vec<&'static str>, +} + +impl IntegrationMetadata { + fn new(id: &'static str) -> Self { + Self { + id, + routes: Vec::new(), + attribute_rewriters: 0, + script_selectors: Vec::new(), + } + } +} + +/// In-memory registry of integrations discovered from settings. +#[derive(Clone, Default)] +pub struct IntegrationRegistry { + inner: Arc, +} + +impl IntegrationRegistry { + /// Build a registry from the provided settings. + pub fn new(settings: &Settings) -> Self { + let mut inner = IntegrationRegistryInner::default(); + + for builder in crate::integrations::builders() { + if let Some(registration) = builder(settings) { + for proxy in registration.proxies { + for route in proxy.routes() { + if inner + .route_map + .insert( + (route.method.clone(), route.path.to_string()), + (proxy.clone(), registration.integration_id), + ) + .is_some() + { + panic!( + "Integration route collision detected for {} {}", + route.method, route.path + ); + } + inner.routes.push((route, registration.integration_id)); + } + } + inner + .html_rewriters + .extend(registration.attribute_rewriters.into_iter()); + inner + .script_rewriters + .extend(registration.script_rewriters.into_iter()); + } + } + + Self { + inner: Arc::new(inner), + } + } + + /// Return true when any proxy is registered for the provided route. + pub fn has_route(&self, method: &Method, path: &str) -> bool { + self.inner + .route_map + .contains_key(&(method.clone(), path.to_string())) + } + + /// Dispatch a proxy request when an integration handles the path. + pub async fn handle_proxy( + &self, + method: &Method, + path: &str, + settings: &Settings, + req: Request, + ) -> Option>> { + if let Some((proxy, _)) = self + .inner + .route_map + .get(&(method.clone(), path.to_string())) + { + Some(proxy.handle(settings, req).await) + } else { + None + } + } + + /// Give integrations a chance to rewrite HTML attributes. + pub fn rewrite_attribute( + &self, + attr_name: &str, + attr_value: &str, + ctx: &IntegrationAttributeContext<'_>, + ) -> AttributeRewriteOutcome { + let mut current = attr_value.to_string(); + let mut changed = false; + for rewriter in &self.inner.html_rewriters { + if !rewriter.handles_attribute(attr_name) { + continue; + } + match rewriter.rewrite(attr_name, ¤t, ctx) { + AttributeRewriteAction::Keep => {} + AttributeRewriteAction::Replace(next_value) => { + current = next_value; + changed = true; + } + AttributeRewriteAction::RemoveElement => { + return AttributeRewriteOutcome::RemoveElement; + } + } + } + + if changed { + AttributeRewriteOutcome::Replaced(current) + } else { + AttributeRewriteOutcome::Unchanged + } + } + + /// Expose registered script/text rewriters for HTML processing. + pub fn script_rewriters(&self) -> Vec> { + self.inner.script_rewriters.clone() + } + + /// Provide a snapshot of registered integrations and their hooks. + pub fn registered_integrations(&self) -> Vec { + let mut map: BTreeMap<&'static str, IntegrationMetadata> = BTreeMap::new(); + + for (route, integration_id) in &self.inner.routes { + let entry = map + .entry(*integration_id) + .or_insert_with(|| IntegrationMetadata::new(integration_id)); + entry + .routes + .push(IntegrationEndpoint::new(route.method.clone(), route.path)); + } + + for rewriter in &self.inner.html_rewriters { + let entry = map + .entry(rewriter.integration_id()) + .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); + entry.attribute_rewriters += 1; + } + + for rewriter in &self.inner.script_rewriters { + let entry = map + .entry(rewriter.integration_id()) + .or_insert_with(|| IntegrationMetadata::new(rewriter.integration_id())); + entry.script_selectors.push(rewriter.selector()); + } + + map.into_values().collect() + } + + #[cfg(test)] + pub fn from_rewriters( + attribute_rewriters: Vec>, + script_rewriters: Vec>, + ) -> Self { + Self { + inner: Arc::new(IntegrationRegistryInner { + route_map: HashMap::new(), + routes: Vec::new(), + html_rewriters: attribute_rewriters, + script_rewriters, + }), + } + } +} diff --git a/crates/common/src/integrations/testlight.rs b/crates/common/src/integrations/testlight.rs new file mode 100644 index 0000000..419bd8f --- /dev/null +++ b/crates/common/src/integrations/testlight.rs @@ -0,0 +1,332 @@ +use std::sync::Arc; + +use async_trait::async_trait; +use error_stack::{Report, ResultExt}; +use fastly::http::{header, Method}; +use fastly::{Request, Response}; +use serde::{Deserialize, Serialize}; +use serde_json::{Map, Value}; +use validator::Validate; + +use crate::backend::ensure_backend_from_url; +use crate::constants::{HEADER_SYNTHETIC_FRESH, HEADER_SYNTHETIC_TRUSTED_SERVER}; +use crate::error::TrustedServerError; +use crate::integrations::{ + AttributeRewriteAction, IntegrationAttributeContext, IntegrationAttributeRewriter, + IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, +}; +use crate::settings::{IntegrationConfig as IntegrationConfigTrait, Settings}; +use crate::synthetic::{generate_synthetic_id, get_or_generate_synthetic_id}; +use crate::tsjs; + +const TESTLIGHT_INTEGRATION_ID: &str = "testlight"; + +#[derive(Debug, Deserialize, Validate)] +pub struct TestlightConfig { + #[serde(default = "default_enabled")] + pub enabled: bool, + #[validate(url)] + pub endpoint: String, + #[serde(default = "default_timeout_ms")] + #[validate(range(min = 10, max = 60000))] + pub timeout_ms: u32, + #[serde(default = "default_shim_src")] + #[validate(length(min = 1))] + pub shim_src: String, + #[serde(default)] + pub rewrite_scripts: bool, +} + +impl IntegrationConfigTrait for TestlightConfig { + fn is_enabled(&self) -> bool { + self.enabled + } +} + +#[derive(Debug, Deserialize, Serialize, Validate)] +struct TestlightRequestBody { + #[validate(nested)] + #[serde(default)] + user: TestlightUserSection, + #[validate(nested)] + #[serde(default)] + imp: Vec, + #[serde(flatten)] + extra: Map, +} + +#[derive(Debug, Default, Deserialize, Serialize, Validate)] +struct TestlightUserSection { + #[serde(default)] + #[validate(length(min = 1))] + id: Option, + #[serde(flatten)] + extra: Map, +} + +#[derive(Debug, Default, Deserialize, Serialize, Validate)] +struct TestlightImp { + #[serde(default)] + #[validate(length(min = 1))] + id: Option, + #[serde(flatten)] + extra: Map, +} + +#[derive(Debug, Deserialize, Serialize)] +struct TestlightResponseBody { + #[serde(flatten)] + fields: Map, +} + +pub struct TestlightIntegration { + config: TestlightConfig, +} + +impl TestlightIntegration { + fn new(config: TestlightConfig) -> Arc { + Arc::new(Self { config }) + } + + fn error(message: impl Into) -> TrustedServerError { + TrustedServerError::Integration { + integration: TESTLIGHT_INTEGRATION_ID.to_string(), + message: message.into(), + } + } +} + +fn build(settings: &Settings) -> Option> { + let config = match settings.integration_config::(TESTLIGHT_INTEGRATION_ID) { + Ok(Some(config)) => config, + Ok(None) => return None, + Err(err) => { + log::error!("Failed to load Testlight integration config: {err:?}"); + return None; + } + }; + + Some(TestlightIntegration::new(config)) +} + +pub fn register(settings: &Settings) -> Option { + let integration = build(settings)?; + Some( + IntegrationRegistration::builder(TESTLIGHT_INTEGRATION_ID) + .with_proxy(integration.clone()) + .with_attribute_rewriter(integration) + .build(), + ) +} + +#[async_trait(?Send)] +impl IntegrationProxy for TestlightIntegration { + fn routes(&self) -> Vec { + vec![IntegrationEndpoint::post("/integrations/testlight/auction")] + } + + async fn handle( + &self, + settings: &Settings, + mut req: Request, + ) -> Result> { + let mut payload = serde_json::from_slice::(&req.take_body_bytes()) + .change_context(Self::error("Failed to parse request body"))?; + payload + .validate() + .map_err(|err| Report::new(Self::error(format!("Invalid request payload: {err}"))))?; + + let synthetic_id = get_or_generate_synthetic_id(settings, &req) + .change_context(Self::error("Failed to fetch or mint synthetic ID"))?; + let fresh_id = generate_synthetic_id(settings, &req) + .change_context(Self::error("Failed to mint fresh synthetic ID"))?; + + payload.user.id = Some(synthetic_id.clone()); + + let mut upstream = Request::new(Method::POST, self.config.endpoint.clone()); + upstream.set_header(header::CONTENT_TYPE, "application/json"); + upstream + .set_body_json(&payload) + .change_context(Self::error("Failed to serialize request body"))?; + + if let Some(user_agent) = req.get_header(header::USER_AGENT) { + upstream.set_header(header::USER_AGENT, user_agent); + } + + let backend = ensure_backend_from_url(&self.config.endpoint) + .change_context(Self::error("Failed to determine backend"))?; + let mut response = upstream + .send(backend) + .change_context(Self::error("Failed to contact upstream integration"))?; + + // Attempt to parse response into structured form for logging/future transforms. + let response_body = response.take_body_bytes(); + match serde_json::from_slice::(&response_body) { + Ok(body) => { + response + .set_body_json(&body) + .change_context(Self::error("Failed to serialize integration response body"))?; + } + Err(_) => { + // Preserve original body if the integration responded with non-JSON content. + response.set_body(response_body); + } + } + + response.set_header(HEADER_SYNTHETIC_TRUSTED_SERVER, &synthetic_id); + response.set_header(HEADER_SYNTHETIC_FRESH, &fresh_id); + Ok(response) + } +} + +impl IntegrationAttributeRewriter for TestlightIntegration { + fn integration_id(&self) -> &'static str { + TESTLIGHT_INTEGRATION_ID + } + + fn handles_attribute(&self, attribute: &str) -> bool { + self.config.rewrite_scripts && matches!(attribute, "src" | "href") + } + + fn rewrite( + &self, + _attr_name: &str, + attr_value: &str, + _ctx: &IntegrationAttributeContext<'_>, + ) -> AttributeRewriteAction { + if !self.config.rewrite_scripts { + return AttributeRewriteAction::keep(); + } + + let lowered = attr_value.to_ascii_lowercase(); + if lowered.contains("testlight.js") { + AttributeRewriteAction::replace(self.config.shim_src.clone()) + } else { + AttributeRewriteAction::keep() + } + } +} + +fn default_timeout_ms() -> u32 { + 1000 +} + +fn default_shim_src() -> String { + // Testlight is included in the unified bundle, so we return the unified script source + tsjs::unified_script_src() +} + +fn default_enabled() -> bool { + true +} + +impl Default for TestlightRequestBody { + fn default() -> Self { + Self { + user: TestlightUserSection::default(), + imp: Vec::new(), + extra: Map::new(), + } + } +} + +impl Default for TestlightResponseBody { + fn default() -> Self { + Self { fields: Map::new() } + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::{test_support::tests::create_test_settings, tsjs}; + use fastly::http::Method; + use serde_json::json; + + #[test] + fn build_requires_config() { + let settings = create_test_settings(); + assert!( + build(&settings).is_none(), + "Should not build without integration config" + ); + } + + #[test] + fn html_rewriter_replaces_integration_script() { + let shim_src = tsjs::unified_script_src(); + let config = TestlightConfig { + enabled: true, + endpoint: "https://example.com/openrtb".to_string(), + timeout_ms: 1000, + shim_src: shim_src.clone(), + rewrite_scripts: true, + }; + let integration = TestlightIntegration::new(config); + + let ctx = IntegrationAttributeContext { + attribute_name: "src", + request_host: "edge.example.com", + request_scheme: "https", + origin_host: "origin.example.com", + }; + + let rewritten = + integration.rewrite("src", "https://cdn.testlight.net/v1/testlight.js", &ctx); + assert!( + matches!( + rewritten, + AttributeRewriteAction::Replace(ref value) if value == &shim_src + ), + "Should swap integration script for trusted shim" + ); + } + + #[test] + fn html_rewriter_is_noop_when_disabled() { + let shim_src = tsjs::unified_script_src(); + let config = TestlightConfig { + enabled: true, + endpoint: "https://example.com/openrtb".to_string(), + timeout_ms: 1000, + shim_src, + rewrite_scripts: false, + }; + let integration = TestlightIntegration::new(config); + let ctx = IntegrationAttributeContext { + attribute_name: "src", + request_host: "edge.example.com", + request_scheme: "https", + origin_host: "origin.example.com", + }; + + assert!(matches!( + integration.rewrite("src", "https://cdn.testlight.net/script.js", &ctx), + AttributeRewriteAction::Keep + )); + } + + #[test] + fn build_uses_settings_integration_block() { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + TESTLIGHT_INTEGRATION_ID.to_string(), + &json!({ + "enabled": true, + "endpoint": "https://example.com/bid", + "rewrite_scripts": true, + }), + ) + .expect("should insert integration config"); + + let integration = build(&settings).expect("Integration should build with config"); + let routes = integration.routes(); + assert!( + routes.iter().any(|route| route.method == Method::POST + && route.path == "/integrations/testlight/auction"), + "Integration should register POST /integrations/testlight/auction" + ); + } +} diff --git a/crates/common/src/lib.rs b/crates/common/src/lib.rs index 124b839..3af42b8 100644 --- a/crates/common/src/lib.rs +++ b/crates/common/src/lib.rs @@ -34,6 +34,7 @@ pub mod fastly_storage; pub mod geo; pub mod html_processor; pub mod http_util; +pub mod integrations; pub mod models; pub mod openrtb; pub mod prebid_proxy; diff --git a/crates/common/src/publisher.rs b/crates/common/src/publisher.rs index 2820d87..a8c13a2 100644 --- a/crates/common/src/publisher.rs +++ b/crates/common/src/publisher.rs @@ -8,6 +8,7 @@ use crate::http_util::serve_static_with_etag; use crate::constants::{HEADER_SYNTHETIC_TRUSTED_SERVER, HEADER_X_COMPRESS_HINT}; use crate::cookies::create_synthetic_cookie; use crate::error::TrustedServerError; +use crate::integrations::IntegrationRegistry; use crate::settings::Settings; use crate::streaming_processor::{Compression, PipelineConfig, StreamProcessor, StreamingPipeline}; use crate::streaming_replacer::create_url_replacer; @@ -105,6 +106,7 @@ struct ProcessResponseParams<'a> { request_scheme: &'a str, settings: &'a Settings, content_type: &'a str, + integration_registry: &'a IntegrationRegistry, } /// Process response body in streaming fashion with compression preservation @@ -129,6 +131,7 @@ fn process_response_streaming( params.request_host, params.request_scheme, params.settings, + params.integration_registry, )?; let config = PipelineConfig { @@ -171,11 +174,17 @@ fn create_html_stream_processor( request_host: &str, request_scheme: &str, settings: &Settings, + integration_registry: &IntegrationRegistry, ) -> Result> { use crate::html_processor::{create_html_processor, HtmlProcessorConfig}; - let config = - HtmlProcessorConfig::from_settings(settings, origin_host, request_host, request_scheme); + let config = HtmlProcessorConfig::from_settings( + settings, + integration_registry, + origin_host, + request_host, + request_scheme, + ); Ok(create_html_processor(config)) } @@ -193,6 +202,7 @@ fn create_html_stream_processor( /// - The origin backend is unreachable pub fn handle_publisher_request( settings: &Settings, + integration_registry: &IntegrationRegistry, mut req: Request, ) -> Result> { log::info!("Proxying request to publisher_origin"); @@ -300,6 +310,7 @@ pub fn handle_publisher_request( request_scheme: &request_scheme, settings, content_type: &content_type, + integration_registry, }; match process_response_streaming(body, params) { Ok(processed_body) => { diff --git a/crates/common/src/settings.rs b/crates/common/src/settings.rs index a9c8964..fdf8a88 100644 --- a/crates/common/src/settings.rs +++ b/crates/common/src/settings.rs @@ -9,6 +9,7 @@ use serde::{ }; use serde_json::Value as JsonValue; use std::collections::HashMap; +use std::ops::{Deref, DerefMut}; use std::sync::OnceLock; use url::Url; use validator::{Validate, ValidationError}; @@ -110,6 +111,107 @@ impl Default for NextJs { } } +#[derive(Debug, Default, Deserialize, Serialize)] +pub struct IntegrationSettings { + #[serde(flatten)] + entries: HashMap, +} + +pub trait IntegrationConfig: DeserializeOwned + Validate { + fn is_enabled(&self) -> bool; +} + +impl IntegrationSettings { + #[cfg_attr(not(test), allow(dead_code))] + pub fn insert_config( + &mut self, + integration_id: impl Into, + value: &T, + ) -> Result<(), Report> + where + T: Serialize, + { + let json = + serde_json::to_value(value).change_context(TrustedServerError::Configuration { + message: "Failed to serialize integration configuration".to_string(), + })?; + self.entries.insert(integration_id.into(), json); + Ok(()) + } + + fn normalize_env_value(value: JsonValue) -> JsonValue { + match value { + JsonValue::Object(map) => JsonValue::Object( + map.into_iter() + .map(|(key, val)| (key, Self::normalize_env_value(val))) + .collect(), + ), + JsonValue::Array(items) => { + JsonValue::Array(items.into_iter().map(Self::normalize_env_value).collect()) + } + JsonValue::String(raw) => { + if let Ok(parsed) = serde_json::from_str::(&raw) { + parsed + } else { + JsonValue::String(raw) + } + } + other => other, + } + } + + pub fn get_typed( + &self, + integration_id: &str, + ) -> Result, Report> + where + T: IntegrationConfig, + { + let raw = match self.entries.get(integration_id) { + Some(value) => value, + None => return Ok(None), + }; + + let normalized = Self::normalize_env_value(raw.clone()); + + let config: T = serde_json::from_value(normalized).change_context( + TrustedServerError::Configuration { + message: format!( + "Integration '{integration_id}' configuration could not be parsed" + ), + }, + )?; + + config.validate().map_err(|err| { + Report::new(TrustedServerError::Configuration { + message: format!( + "Integration '{integration_id}' configuration failed validation: {err}" + ), + }) + })?; + + if !config.is_enabled() { + return Ok(None); + } + + Ok(Some(config)) + } +} + +impl Deref for IntegrationSettings { + type Target = HashMap; + + fn deref(&self) -> &Self::Target { + &self.entries + } +} + +impl DerefMut for IntegrationSettings { + fn deref_mut(&mut self) -> &mut Self::Target { + &mut self.entries + } +} + fn deserialize_nextjs_attributes<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -188,6 +290,8 @@ pub struct Settings { #[serde(default)] #[validate(nested)] pub synthetic: Synthetic, + #[serde(default)] + pub integrations: IntegrationSettings, #[serde(default, deserialize_with = "vec_from_seq_or_map")] #[validate(nested)] pub handlers: Vec, @@ -260,6 +364,16 @@ impl Settings { .iter() .find(|handler| handler.matches_path(path)) } + + pub fn integration_config( + &self, + integration_id: &str, + ) -> Result, Report> + where + T: IntegrationConfig, + { + self.integrations.get_typed(integration_id) + } } fn validate_path(value: &str) -> Result<(), ValidationError> { @@ -333,7 +447,7 @@ mod tests { use super::*; use regex::Regex; - use crate::test_support::tests::crate_test_settings_str; + use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; #[test] fn test_settings_new() { @@ -587,10 +701,6 @@ mod tests { #[test] fn test_set_env() { - let re = Regex::new(r"ad_partner_url = .*").unwrap(); - let toml_str = crate_test_settings_str(); - let toml_str = re.replace(&toml_str, ""); - temp_env::with_var( format!( "{}{}PUBLISHER{}ORIGIN_URL", @@ -600,7 +710,7 @@ mod tests { ), Some("https://change-publisher.com"), || { - let settings = Settings::from_toml(&toml_str); + let settings = Settings::from_toml(&crate_test_settings_str()); assert!(settings.is_ok(), "Settings should load from embedded TOML"); assert_eq!( @@ -697,4 +807,90 @@ mod tests { }; assert_eq!(publisher.origin_host(), "[::1]:8080"); } + + #[test] + fn test_integration_settings_from_env() { + use crate::integrations::testlight::TestlightConfig; + + let toml_str = crate_test_settings_str(); + + let origin_key = format!( + "{}{}PUBLISHER{}ORIGIN_URL", + ENVIRONMENT_VARIABLE_PREFIX, + ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR + ); + + let integration_prefix = format!( + "{}{}INTEGRATIONS{}TESTLIGHT{}", + ENVIRONMENT_VARIABLE_PREFIX, + ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR + ); + + let endpoint_key = format!("{}ENDPOINT", integration_prefix); + let timeout_key = format!("{}TIMEOUT_MS", integration_prefix); + let rewrite_key = format!("{}REWRITE_SCRIPTS", integration_prefix); + let enabled_key = format!("{}ENABLED", integration_prefix); + + temp_env::with_var( + origin_key, + Some("https://origin.test-publisher.com"), + || { + temp_env::with_var( + endpoint_key, + Some("https://testlight-env.test/auction"), + || { + temp_env::with_var(timeout_key, Some("2500"), || { + temp_env::with_var(rewrite_key, Some("true"), || { + temp_env::with_var(enabled_key, Some("true"), || { + let settings = Settings::from_toml(&toml_str) + .expect("Settings should load"); + + let config = settings + .integration_config::("testlight") + .expect("integration parsing should succeed") + .expect("integration should be enabled"); + + assert_eq!( + config.endpoint, + "https://testlight-env.test/auction" + ); + assert_eq!(config.timeout_ms, 2500); + assert!(config.rewrite_scripts); + assert!(config.enabled); + }); + }); + }); + }, + ); + }, + ); + } + + #[test] + fn test_disabled_integration_does_not_register() { + use crate::integrations::testlight::TestlightConfig; + use serde_json::json; + + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + "testlight", + &json!({ + "enabled": false, + "endpoint": "https://testlight.test/auction", + "rewrite_scripts": true, + }), + ) + .expect("should insert integration config"); + + let config = settings + .integration_config::("testlight") + .expect("integration parsing should succeed"); + + assert!(config.is_none(), "Disabled integrations should be skipped"); + } } diff --git a/crates/common/src/tsjs.rs b/crates/common/src/tsjs.rs index c4c4cef..26690da 100644 --- a/crates/common/src/tsjs.rs +++ b/crates/common/src/tsjs.rs @@ -21,27 +21,12 @@ fn script_tag_for(bundle: TsjsBundle, attrs: &str) -> String { ) } -/// ` - // optionally load Prebid shim when pbjs is present - // - -``` - -## Auto‑Rewrite (Server) - -- When auto-configure is enabled, the HTML processor injects the core loader and rewrites any Prebid script URLs to `/static/tsjs=tsjs-ext.min.js`. The extension aliases `window.pbjs` to `window.tsjs` and flushes `pbjs.que`. -- Proxied creative HTML injects the creative helper once at the top of ``: `/static/tsjs=tsjs-creative.min.js`. The helper monitors anchors for script-driven rewrites and rebuilds first-party click URLs whenever creatives mutate them. - -## First-Party Proxy Flows - -The Rust services (`trusted-server-common`) expose several proxy entry points that work together to keep all ad traffic on the publisher’s domain while propagating the synthetic identifier generated for the user. - -### Publisher Origin Proxy - -- Endpoint: `handle_publisher_request` (`crates/common/src/publisher.rs`). -- Retrieves or generates the trusted synthetic identifier before Fastly consumes the request body. -- Always stamps the proxied response with `X-Synthetic-Fresh` and `x-psid-ts` headers and, when the browser does not already present one, sets the `synthetic_id=` cookie (Secure + SameSite=Lax) bound to the configured publisher domain. -- Result: downstream assets fetched through the same first-party origin automatically include the synthetic ID header/cookie so subsequent proxy layers can read it. - -### Creative Asset Proxy - -- Endpoint: `handle_first_party_proxy` (`crates/common/src/proxy.rs`). -- Accepts the signed `/first-party/proxy?tsurl=...` URLs injected by the HTML rewriter and streams the creative from the third-party origin. -- Extracts the synthetic ID from the inbound cookie or header and forwards it to the creative origin by appending `synthetic_id=` to the rewritten target URL (while preserving existing query parameters). -- Follows HTTP redirects (301/302/303/307/308) up to four hops, re-validating each `Location`, switching to `GET` after 303 responses, and propagating the synthetic ID on every hop. -- Ensures the response body is rewritten when it is HTML/CSS/JS so all nested asset requests loop back through the same first-party proxy. - -### Click-Through Proxy - -- Endpoint: `handle_first_party_click` (`crates/common/src/proxy.rs`). -- Validates the signed `/first-party/click` URL generated for anchors inside proxied creatives. -- On success, issues an HTTP 302 to the reconstructed destination and appends `synthetic_id=` if the user presented one, letting downstream measurement end points correlate the click with the original synthetic identifier. -- Ensures click responses are never cached (`Cache-Control: no-store, private`). - -Together these layers guarantee that the synthetic identifier generated on the publisher response is preserved throughout page loads, asset fetches, and click-throughs without exposing the third-party origins directly to the browser. - -## Notes - -- By default, the build fails if `tsjs-core.js` cannot be produced. To change behavior: - - `TSJS_SKIP_BUILD=1`: skip running npm; requires `dist/tsjs-core.js` to exist so it can be copied to `OUT_DIR`. - - `TSJS_ALLOW_FALLBACK=1`: allow using a checked‑in `dist/tsjs-core.js` if the npm build didn’t produce an output. - - `TSJS_TEST=1`: run `npm test` during the build. diff --git a/crates/js/build.rs b/crates/js/build.rs index 6a4d3ae..da9fbe3 100644 --- a/crates/js/build.rs +++ b/crates/js/build.rs @@ -3,25 +3,7 @@ use std::fs; use std::path::{Path, PathBuf}; use std::process::Command; -struct BundleSpec { - filename: &'static str, - required: bool, -} - -const BUNDLES: &[BundleSpec] = &[ - BundleSpec { - filename: "tsjs-core.js", - required: true, - }, - BundleSpec { - filename: "tsjs-ext.js", - required: false, - }, - BundleSpec { - filename: "tsjs-creative.js", - required: false, - }, -]; +const UNIFIED_BUNDLE: &str = "tsjs-unified.js"; fn main() { // Rebuild if TS sources change (belt-and-suspenders): enumerate every file under ts/ @@ -78,11 +60,15 @@ fn main() { .status(); } - // Build bundle + // Build unified bundle if !skip { if let Some(npm_path) = npm.clone() { - let status = Command::new(npm_path) - .args(["run", "build"]) + println!("cargo:warning=tsjs: Building unified bundle"); + let js_modules = env::var("TSJS_MODULES").unwrap_or("".to_string()); + + let status = Command::new(&npm_path) + .env("TSJS_MODULES", js_modules) + .args(["run", "build:custom"]) .current_dir(&ts_dir) .status(); if !status.as_ref().map(|s| s.success()).unwrap_or(false) { @@ -91,21 +77,19 @@ fn main() { } } - // Copy the result into OUT_DIR for include_str! - for bundle in BUNDLES { - copy_bundle(bundle, &crate_dir, &dist_dir, &out_dir); - } + // Copy unified bundle into OUT_DIR for include_str! + copy_bundle(UNIFIED_BUNDLE, true, &crate_dir, &dist_dir, &out_dir); } -fn copy_bundle(spec: &BundleSpec, crate_dir: &Path, dist_dir: &Path, out_dir: &Path) { - let primary = dist_dir.join(spec.filename); - let fallback = crate_dir.join("dist").join(spec.filename); - let target = out_dir.join(spec.filename); +fn copy_bundle(filename: &str, required: bool, crate_dir: &Path, dist_dir: &Path, out_dir: &Path) { + let primary = dist_dir.join(filename); + let fallback = crate_dir.join("dist").join(filename); + let target = out_dir.join(filename); for source in [&primary, &fallback] { if source.exists() { if let Err(e) = fs::copy(source, &target) { - if spec.required { + if required { panic!("tsjs: failed to copy {:?} to {:?}: {}", source, target, e); } } @@ -113,10 +97,10 @@ fn copy_bundle(spec: &BundleSpec, crate_dir: &Path, dist_dir: &Path, out_dir: &P } } - if spec.required { + if required { panic!( "tsjs: bundle {} not found: {:?} (and fallback {:?}). Ensure Node is installed and `npm run build` succeeds, or commit dist/{}.", - spec.filename, primary, fallback, spec.filename + filename, primary, fallback, filename ); } diff --git a/crates/js/lib/.gitignore b/crates/js/lib/.gitignore new file mode 100644 index 0000000..f1283e5 --- /dev/null +++ b/crates/js/lib/.gitignore @@ -0,0 +1,2 @@ +# Auto-generated files +src/generated-modules.ts diff --git a/crates/js/lib/package-lock.json b/crates/js/lib/package-lock.json index 6e0c2d6..33c2aa4 100644 --- a/crates/js/lib/package-lock.json +++ b/crates/js/lib/package-lock.json @@ -166,7 +166,6 @@ } ], "license": "MIT", - "peer": true, "engines": { "node": ">=18" }, @@ -210,7 +209,6 @@ } ], "license": "MIT", - "peer": true, "engines": { "node": ">=18" } @@ -1409,7 +1407,6 @@ "integrity": "sha512-tK3GPFWbirvNgsNKto+UmB/cRtn6TZfyw0D6IKrW55n6Vbs7KJoZtI//kpTKzE/DUmmnAFD8/Ca46s7Obs92/w==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@typescript-eslint/scope-manager": "8.46.4", "@typescript-eslint/types": "8.46.4", @@ -1726,7 +1723,6 @@ "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", "dev": true, "license": "MIT", - "peer": true, "bin": { "acorn": "bin/acorn" }, @@ -2032,7 +2028,6 @@ } ], "license": "MIT", - "peer": true, "dependencies": { "baseline-browser-mapping": "^2.8.25", "caniuse-lite": "^1.0.30001754", @@ -2720,7 +2715,6 @@ "integrity": "sha512-BhHmn2yNOFA9H9JmmIVKJmd288g9hrVRDkdoIgRCRuSySRUHH7r/DI6aAXW9T1WwUuY3DFgrcaqB+deURBLR5g==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@eslint-community/eslint-utils": "^4.8.0", "@eslint-community/regexpp": "^4.12.1", @@ -5529,7 +5523,6 @@ "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==", "dev": true, "license": "MIT", - "peer": true, "engines": { "node": ">=12" }, @@ -5746,7 +5739,6 @@ "integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==", "dev": true, "license": "Apache-2.0", - "peer": true, "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" @@ -5945,7 +5937,6 @@ "integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==", "dev": true, "license": "MIT", - "peer": true, "engines": { "node": ">=12" }, diff --git a/crates/js/lib/package.json b/crates/js/lib/package.json index 8235573..93164d3 100644 --- a/crates/js/lib/package.json +++ b/crates/js/lib/package.json @@ -5,8 +5,9 @@ "type": "module", "description": "Trusted Server tsjs TypeScript library with queue and simple banner rendering.", "scripts": { - "build": "vite build --mode core && vite build --mode ext && vite build --mode creative", - "dev": "node scripts/watch-all.mjs", + "build": "vite build", + "build:custom": "vite build", + "dev": "vite build --watch", "test": "vitest run", "test:watch": "vitest", "lint": "eslint \"src/**/*.{ts,tsx}\"", diff --git a/crates/js/lib/scripts/watch-all.mjs b/crates/js/lib/scripts/watch-all.mjs deleted file mode 100755 index 6434afe..0000000 --- a/crates/js/lib/scripts/watch-all.mjs +++ /dev/null @@ -1,33 +0,0 @@ -#!/usr/bin/env node -import { spawn } from 'node:child_process'; - -const modes = ['core', 'ext', 'creative']; -const children = []; - -function spawnWatcher(mode) { - const child = spawn('npx', ['vite', 'build', '--mode', mode, '--watch'], { - stdio: 'inherit', - env: process.env, - }); - child.on('exit', (code, signal) => { - if (signal) { - console.log(`[watch] mode ${mode} exited via signal ${signal}`); - } else { - console.log(`[watch] mode ${mode} exited with code ${code}`); - } - }); - children.push(child); -} - -modes.forEach(spawnWatcher); - -function shutdown(signal) { - console.log(`\n[watch] received ${signal}; shutting down watchers...`); - for (const child of children) { - if (!child.killed) child.kill('SIGTERM'); - } - process.exit(0); -} - -process.on('SIGINT', () => shutdown('SIGINT')); -process.on('SIGTERM', () => shutdown('SIGTERM')); diff --git a/crates/js/lib/src/index.ts b/crates/js/lib/src/index.ts new file mode 100644 index 0000000..bfe6e69 --- /dev/null +++ b/crates/js/lib/src/index.ts @@ -0,0 +1,41 @@ +// Unified tsjs bundle entry point +// This file conditionally imports modules based on build-time configuration +import { modules, type ModuleName } from './generated-modules'; +import { log } from './core/log'; + +const VERSION = '0.1.0-unified'; + +// Log which modules are included in this build +const includedModules = Object.keys(modules) as ModuleName[]; +log.info('tsjs unified bundle initialized', { + version: VERSION, + modules: includedModules, +}); + +// The core module sets up the main API and should always be included +// If core is included, it will have already initialized the tsjs global +// and set up the queue system + +// Initialize optional modules if they're included +for (const [moduleName, moduleExports] of Object.entries(modules)) { + if (moduleName === 'core') { + // Core is already initialized via its own IIFE-style init code + continue; + } + + // For other modules, check if they have an init function or are self-initializing + if (typeof moduleExports === 'object' && moduleExports !== null) { + // Log that the module is available + log.debug(`tsjs: module '${moduleName}' loaded`); + + // Some modules like 'ext' are self-initializing (they run on import) + // Some modules like 'creative' export an API object + // We don't need to do anything special here - just importing them is enough + } +} + +// Re-export core types for convenience +export type { AdUnit, TsjsApi } from './core/types'; + +// Export the modules object for advanced use cases +export { modules }; diff --git a/crates/js/lib/src/creative/click.ts b/crates/js/lib/src/integrations/creative/click.ts similarity index 98% rename from crates/js/lib/src/creative/click.ts rename to crates/js/lib/src/integrations/creative/click.ts index dc5289a..a505c03 100644 --- a/crates/js/lib/src/creative/click.ts +++ b/crates/js/lib/src/integrations/creative/click.ts @@ -1,8 +1,8 @@ // Click guard runtime: detects mutated tracking URLs and rebuilds signed first-party clicks. -import { log } from '../core/log'; -import { creativeGlobal } from '../shared/globals'; -import { delay, queueTask } from '../shared/async'; -import { createMutationScheduler } from '../shared/scheduler'; +import { log } from '../../core/log'; +import { creativeGlobal } from '../../shared/globals'; +import { delay, queueTask } from '../../shared/async'; +import { createMutationScheduler } from '../../shared/scheduler'; type AnchorLike = HTMLAnchorElement | HTMLAreaElement; type Canon = { base: string; params: Record }; diff --git a/crates/js/lib/src/creative/dynamic_src_guard.ts b/crates/js/lib/src/integrations/creative/dynamic_src_guard.ts similarity index 99% rename from crates/js/lib/src/creative/dynamic_src_guard.ts rename to crates/js/lib/src/integrations/creative/dynamic_src_guard.ts index 569e0ef..2fc216c 100644 --- a/crates/js/lib/src/creative/dynamic_src_guard.ts +++ b/crates/js/lib/src/integrations/creative/dynamic_src_guard.ts @@ -1,5 +1,5 @@ -import { log } from '../core/log'; -import { createMutationScheduler } from '../shared/scheduler'; +import { log } from '../../core/log'; +import { createMutationScheduler } from '../../shared/scheduler'; type ElementWithSrc = Element & { src: string }; diff --git a/crates/js/lib/src/creative/iframe.ts b/crates/js/lib/src/integrations/creative/iframe.ts similarity index 100% rename from crates/js/lib/src/creative/iframe.ts rename to crates/js/lib/src/integrations/creative/iframe.ts diff --git a/crates/js/lib/src/creative/image.ts b/crates/js/lib/src/integrations/creative/image.ts similarity index 100% rename from crates/js/lib/src/creative/image.ts rename to crates/js/lib/src/integrations/creative/image.ts diff --git a/crates/js/lib/src/creative/index.ts b/crates/js/lib/src/integrations/creative/index.ts similarity index 95% rename from crates/js/lib/src/creative/index.ts rename to crates/js/lib/src/integrations/creative/index.ts index 2f16bbb..3955535 100644 --- a/crates/js/lib/src/creative/index.ts +++ b/crates/js/lib/src/integrations/creative/index.ts @@ -1,7 +1,7 @@ // Entry point for the creative runtime: wires up click + image + iframe guards globally. -import { log } from '../core/log'; -import type { TsCreativeConfig, CreativeWindow, TsCreativeApi } from '../shared/globals'; -import { creativeGlobal, resolveWindow } from '../shared/globals'; +import { log } from '../../core/log'; +import type { TsCreativeConfig, CreativeWindow, TsCreativeApi } from '../../shared/globals'; +import { creativeGlobal, resolveWindow } from '../../shared/globals'; import { installClickGuard } from './click'; import { installDynamicImageProxy } from './image'; diff --git a/crates/js/lib/src/creative/proxy_sign.ts b/crates/js/lib/src/integrations/creative/proxy_sign.ts similarity index 97% rename from crates/js/lib/src/creative/proxy_sign.ts rename to crates/js/lib/src/integrations/creative/proxy_sign.ts index fc89acc..b18d7d1 100644 --- a/crates/js/lib/src/creative/proxy_sign.ts +++ b/crates/js/lib/src/integrations/creative/proxy_sign.ts @@ -1,4 +1,4 @@ -import { log } from '../core/log'; +import { log } from '../../core/log'; const PROXY_PREFIX = '/first-party/proxy'; diff --git a/crates/js/lib/src/ext/index.ts b/crates/js/lib/src/integrations/ext/index.ts similarity index 100% rename from crates/js/lib/src/ext/index.ts rename to crates/js/lib/src/integrations/ext/index.ts diff --git a/crates/js/lib/src/ext/prebidjs.ts b/crates/js/lib/src/integrations/ext/prebidjs.ts similarity index 91% rename from crates/js/lib/src/ext/prebidjs.ts rename to crates/js/lib/src/integrations/ext/prebidjs.ts index 4d6224c..242ebc0 100644 --- a/crates/js/lib/src/ext/prebidjs.ts +++ b/crates/js/lib/src/integrations/ext/prebidjs.ts @@ -1,9 +1,14 @@ // Prebid.js compatibility shim: exposes tsjs API through the legacy pbjs global. -import type { TsjsApi, HighestCpmBid, RequestAdsCallback, RequestAdsOptions } from '../core/types'; -import { log } from '../core/log'; -import { installQueue } from '../core/queue'; -import { getAllCodes, getAllUnits, firstSize } from '../core/registry'; -import { resolvePrebidWindow, PrebidWindow } from '../shared/globals'; +import type { + TsjsApi, + HighestCpmBid, + RequestAdsCallback, + RequestAdsOptions, +} from '../../core/types'; +import { log } from '../../core/log'; +import { installQueue } from '../../core/queue'; +import { getAllCodes, getAllUnits, firstSize } from '../../core/registry'; +import { resolvePrebidWindow, PrebidWindow } from '../../shared/globals'; type RequestBidsFunction = ( callbackOrOpts?: RequestAdsCallback | RequestAdsOptions, opts?: RequestAdsOptions diff --git a/crates/js/lib/src/ext/types.ts b/crates/js/lib/src/integrations/ext/types.ts similarity index 100% rename from crates/js/lib/src/ext/types.ts rename to crates/js/lib/src/integrations/ext/types.ts diff --git a/crates/js/lib/src/integrations/testlight/index.ts b/crates/js/lib/src/integrations/testlight/index.ts new file mode 100644 index 0000000..3db00ce --- /dev/null +++ b/crates/js/lib/src/integrations/testlight/index.ts @@ -0,0 +1,81 @@ +import type { TsjsApi } from '../../core/types'; +import { installQueue } from '../../core/queue'; +import { log } from '../../core/log'; +import { resolvePrebidWindow, PrebidWindow } from '../../shared/globals'; + +type TestlightCallback = () => void; + +type TestlightGlobal = { + que?: TestlightCallback[]; +}; + +type TestlightWindow = PrebidWindow & { + testlight?: TestlightGlobal; +}; + +function ensureTsjsApi(win: TestlightWindow): TsjsApi { + if (win.tsjs) return win.tsjs; + const stub: TsjsApi = { + version: '0.0.0', + que: [], + addAdUnits: () => undefined, + renderAdUnit: () => undefined, + renderAllAdUnits: () => undefined, + }; + win.tsjs = stub; + return stub; +} + +function installTestlightQueue(api: TsjsApi, win: TestlightWindow): void { + if (!Array.isArray(api.que)) { + installQueue(api, win); + } +} + +function flushCallbacks(queue: TestlightCallback[], api: TsjsApi): void { + while (queue.length > 0) { + const fn = queue.shift(); + if (typeof fn !== 'function') { + continue; + } + try { + if (Array.isArray(api.que)) { + api.que.push(fn); + } else { + fn.call(api); + } + log.debug('testlight shim: flushed callback'); + } catch (err) { + log.debug('testlight shim: queued callback threw', err); + } + } +} + +export function installTestlightShim(): boolean { + const win = resolvePrebidWindow() as TestlightWindow; + const api = ensureTsjsApi(win); + installTestlightQueue(api, win); + + const testlight = (win.testlight = win.testlight ?? {}); + const pending: TestlightCallback[] = Array.isArray(testlight.que) ? [...testlight.que] : []; + const queue: TestlightCallback[] = []; + testlight.que = queue; + + const originalPush = queue.push.bind(queue); + queue.push = function (...callbacks: TestlightCallback[]): number { + const len = originalPush(...callbacks); + flushCallbacks(queue, api); + return len; + }; + + if (pending.length > 0) { + queue.push(...pending); + } + + log.info('testlight shim installed', { queuedCallbacks: queue.length }); + return true; +} + +if (typeof window !== 'undefined') { + installTestlightShim(); +} diff --git a/crates/js/lib/test/creative/click.test.ts b/crates/js/lib/test/integrations/creative/click.test.ts similarity index 100% rename from crates/js/lib/test/creative/click.test.ts rename to crates/js/lib/test/integrations/creative/click.test.ts diff --git a/crates/js/lib/test/creative/helpers.ts b/crates/js/lib/test/integrations/creative/helpers.ts similarity index 88% rename from crates/js/lib/test/creative/helpers.ts rename to crates/js/lib/test/integrations/creative/helpers.ts index 885d4e6..176a5e5 100644 --- a/crates/js/lib/test/creative/helpers.ts +++ b/crates/js/lib/test/integrations/creative/helpers.ts @@ -17,7 +17,7 @@ export const MUTATED_CLICK = 'https://example.com/landing?bar=2'; export const PROXY_RESPONSE = '/first-party/click?tsurl=https%3A%2F%2Fexample.com%2Flanding&bar=2&tstoken=newtoken'; -import type { TsCreativeConfig } from '../../src/shared/globals'; +import type { TsCreativeConfig } from '../../../src/shared/globals'; export async function importCreativeModule(config?: TsCreativeConfig): Promise { const globalRef = globalThis as { @@ -28,7 +28,7 @@ export async function importCreativeModule(config?: TsCreativeConfig): Promise s.trim()) + .filter(Boolean); + + // Discover integration modules: directories in src/integrations/ with index.ts + const integrationModules = fs.existsSync(integrationsDir) + ? fs.readdirSync(integrationsDir).filter((name) => { + const fullPath = path.join(integrationsDir, name); + const stat = fs.statSync(fullPath); + return stat.isDirectory() && fs.existsSync(path.join(fullPath, 'index.ts')); + }) + : []; + + // Always include core first + const finalModules = ['core']; + + // Add requested integrations based on TSJS_MODULES env var + if (requestedModules.length === 0) { + // TSJS_MODULES not set or empty: include all discovered integrations + finalModules.push(...integrationModules); + } else { + // TSJS_MODULES set to list: include only requested integrations (excluding 'core' as it's always added) + const requestedIntegrations = requestedModules.filter((m) => m !== 'core'); + finalModules.push(...integrationModules.filter((m) => requestedIntegrations.includes(m))); + } + + // Generate import statements + // Use namespace imports to capture all exports from each module + const importLines: string[] = []; + const exportEntries: string[] = []; + + for (const moduleName of finalModules) { + if (moduleName === 'core') { + importLines.push(`import * as core from './core/index';`); + } else { + importLines.push(`import * as ${moduleName} from './integrations/${moduleName}/index';`); + } + exportEntries.push(` ${moduleName},`); + } + + // Write the generated file + const output = `// Auto-generated by vite-plugin-tsjs-module-discovery +// DO NOT EDIT - This file is generated at build time based on TSJS_MODULES env var +${importLines.join('\n')} -const BUNDLES: Record = { - core: { - input: path.resolve(__dirname, 'src/core/index.ts'), - fileName: 'tsjs-core.js', - name: 'tsjs', - }, - ext: { - input: path.resolve(__dirname, 'src/ext/index.ts'), - fileName: 'tsjs-ext.js', - name: 'tsjs', - extend: true, - }, - creative: { - input: path.resolve(__dirname, 'src/creative/index.ts'), - fileName: 'tsjs-creative.js', - name: 'tscreative', - }, +export const modules = { +${exportEntries.join('\n')} }; -function resolveBundleName(mode: string | undefined): BundleName { - const fromEnv = process.env.TSJS_BUNDLE?.toLowerCase(); - if (fromEnv && isBundleName(fromEnv)) return fromEnv; +export type ModuleName = ${finalModules.map((m) => `'${m}'`).join(' | ')}; +`; - const normalized = mode?.toLowerCase(); - if (normalized && isBundleName(normalized)) return normalized; - return 'core'; -} + const generatedFilePath = path.join(srcDir, 'generated-modules.ts'); + fs.writeFileSync(generatedFilePath, output); -function isBundleName(value: string): value is BundleName { - return Object.hasOwn(BUNDLES, value); + console.log('[tsjs-module-discovery] Generated src/generated-modules.ts'); + console.log('[tsjs-module-discovery] Discovered integrations:', integrationModules); + console.log('[tsjs-module-discovery] Included modules:', finalModules); + }, + }; } -export default defineConfig(({ mode }) => { - const bundleName = resolveBundleName(mode); - const bundle = BUNDLES[bundleName]; +export default defineConfig(() => { const distDir = path.resolve(__dirname, '../dist'); const buildTimestamp = new Date().toISOString(); const banner = `// build: ${buildTimestamp}\n`; @@ -59,32 +95,17 @@ export default defineConfig(({ mode }) => { sourcemap: false, minify: 'esbuild', rollupOptions: { - input: bundle.input, + input: path.resolve(__dirname, 'src/index.ts'), output: { format: 'iife', dir: distDir, - entryFileNames: bundle.fileName, + entryFileNames: 'tsjs-unified.js', inlineDynamicImports: true, - extend: bundle.extend ?? false, - name: bundle.name, + extend: false, + name: 'tsjs', }, }, }, - plugins: [createTimestampBannerPlugin(banner)], + plugins: [createModuleDiscoveryPlugin()], }; }); - -function createTimestampBannerPlugin(banner: string): Plugin { - return { - name: 'tsjs-build-timestamp-banner', - generateBundle(_options, bundleOutput) { - for (const file of Object.values(bundleOutput)) { - if (file.type === 'chunk') { - file.code = `${banner}${file.code}`; - } else if (file.type === 'asset' && typeof file.source === 'string') { - file.source = `${banner}${file.source}`; - } - } - }, - }; -} diff --git a/crates/js/src/bundle.rs b/crates/js/src/bundle.rs index 8fe02ab..ef29249 100644 --- a/crates/js/src/bundle.rs +++ b/crates/js/src/bundle.rs @@ -1,4 +1,3 @@ -use super::macros::{count_variants, define_tsjs_bundles}; use hex::encode; use sha2::{Digest, Sha256}; @@ -14,11 +13,57 @@ impl TsjsMeta { } } -define_tsjs_bundles!( - Core => "tsjs-core.js", - Ext => "tsjs-ext.js", - Creative => "tsjs-creative.js", -); +const TSJS_BUNDLE_COUNT: usize = 1; + +#[repr(usize)] +#[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] +pub enum TsjsBundle { + Unified, +} + +const METAS: [TsjsMeta; TSJS_BUNDLE_COUNT] = [TsjsMeta::new( + "tsjs-unified.js", + include_str!(concat!(env!("OUT_DIR"), "/tsjs-unified.js")), +)]; + +const ALL_BUNDLES: [TsjsBundle; TSJS_BUNDLE_COUNT] = [TsjsBundle::Unified]; + +impl TsjsBundle { + pub const COUNT: usize = TSJS_BUNDLE_COUNT; + + pub const fn filename(self) -> &'static str { + METAS[self as usize].filename + } + + pub fn minified_filename(self) -> String { + let base = self.filename(); + match base.strip_suffix(".js") { + Some(stem) => format!("{stem}.min.js"), + None => format!("{base}.min.js"), + } + } + + pub(crate) const fn bundle(self) -> &'static str { + METAS[self as usize].bundle + } + + pub(crate) fn filename_map() -> &'static std::collections::HashMap<&'static str, TsjsBundle> { + static MAP: std::sync::OnceLock> = + std::sync::OnceLock::new(); + + MAP.get_or_init(|| { + ALL_BUNDLES + .iter() + .copied() + .map(|bundle| (bundle.filename(), bundle)) + .collect::>() + }) + } + + pub fn from_filename(name: &str) -> Option { + Self::filename_map().get(name).copied() + } +} pub fn bundle_hash(bundle: TsjsBundle) -> String { hash_bundle(bundle.bundle()) @@ -28,6 +73,10 @@ pub fn bundle_for_filename(name: &str) -> Option<&'static str> { TsjsBundle::from_filename(name).map(|bundle| bundle.bundle()) } +pub fn bundle_hash_for_filename(name: &str) -> Option { + TsjsBundle::from_filename(name).map(|bundle| hash_bundle(bundle.bundle())) +} + fn hash_bundle(bundle: &'static str) -> String { let mut hasher = Sha256::new(); hasher.update(bundle.as_bytes()); diff --git a/crates/js/src/lib.rs b/crates/js/src/lib.rs index 0d5b497..7da69fc 100644 --- a/crates/js/src/lib.rs +++ b/crates/js/src/lib.rs @@ -1,4 +1,3 @@ pub mod bundle; -mod macros; -pub use bundle::{bundle_for_filename, bundle_hash, TsjsBundle}; +pub use bundle::{bundle_for_filename, bundle_hash, bundle_hash_for_filename, TsjsBundle}; diff --git a/crates/js/src/macros.rs b/crates/js/src/macros.rs deleted file mode 100644 index cd9ea21..0000000 --- a/crates/js/src/macros.rs +++ /dev/null @@ -1,68 +0,0 @@ -macro_rules! count_variants { - ($($variant:ident),+ $(,)?) => { - <[()]>::len(&[$(count_variants!(@unit $variant)),+]) - }; - (@unit $variant:ident) => { () }; -} - -macro_rules! define_tsjs_bundles { - ($($variant:ident => $file:expr),+ $(,)?) => { - const TSJS_BUNDLE_COUNT: usize = count_variants!($($variant),+); - - #[repr(usize)] - #[derive(Copy, Clone, Debug, Eq, PartialEq, Hash)] - pub enum TsjsBundle { - $( $variant ),+ - } - - const METAS: [TsjsMeta; TSJS_BUNDLE_COUNT] = [ - $(TsjsMeta::new($file, include_str!(concat!(env!("OUT_DIR"), "/", $file)))),+ - ]; - - const ALL_BUNDLES: [TsjsBundle; TSJS_BUNDLE_COUNT] = [ - $(TsjsBundle::$variant),+ - ]; - - impl TsjsBundle { - pub const COUNT: usize = TSJS_BUNDLE_COUNT; - - pub const fn filename(self) -> &'static str { - METAS[self as usize].filename - } - - pub fn minified_filename(self) -> String { - let base = self.filename(); - match base.strip_suffix(".js") { - Some(stem) => format!("{stem}.min.js"), - None => format!("{base}.min.js"), - } - } - - pub(crate) const fn bundle(self) -> &'static str { - METAS[self as usize].bundle - } - - pub(crate) fn filename_map( - ) -> &'static ::std::collections::HashMap<&'static str, TsjsBundle> { - static MAP: ::std::sync::OnceLock< - ::std::collections::HashMap<&'static str, TsjsBundle>, - > = ::std::sync::OnceLock::new(); - - MAP.get_or_init(|| { - ALL_BUNDLES - .iter() - .copied() - .map(|bundle| (bundle.filename(), bundle)) - .collect::<::std::collections::HashMap<_, _>>() - }) - } - - pub fn from_filename(name: &str) -> Option { - Self::filename_map().get(name).copied() - } - } - }; -} - -pub(crate) use count_variants; -pub(crate) use define_tsjs_bundles; diff --git a/docs/integration_guide.md b/docs/integration_guide.md new file mode 100644 index 0000000..0f2fee6 --- /dev/null +++ b/docs/integration_guide.md @@ -0,0 +1,251 @@ +# Integration Guide + +This document explains how to integrate a new integration module with the Trusted Server +runtime. The workflow mirrors the built‑in `testlight` sample in +`crates/common/src/integrations/testlight.rs`. + +## Architecture Overview + +| Component | Purpose | +| ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `crates/common/src/integrations/registry.rs` | Defines the `IntegrationProxy`, `IntegrationAttributeRewriter`, and `IntegrationScriptRewriter` traits and hosts the `IntegrationRegistry`, which drives proxy routing and HTML/text rewrites. | +| `Settings::integrations` (`crates/common/src/settings.rs`) | Free‑form JSON blob keyed by integration ID. Use `IntegrationSettings::insert_config` to seed configs; each module deserializes and validates (`validator::Validate`) its own config and exposes an `enabled` flag so the core settings schema stays stable. | +| Fastly entrypoint (`crates/fastly/src/main.rs`) | Instantiates the registry once per request, routes `/integrations//…` requests to the appropriate proxy, and passes the registry to the publisher origin proxy so HTML rewriting remains integration-aware. | +| `html_processor.rs` | Applies first‑party URL rewrites, injects the Trusted Server JS shim, and lets integrations override attribute values (for example to swap script URLs). | + +## Step-by-Step Integration + +### 1. Define integration configuration + +Add a `trusted-server.toml` block and any environment overrides under +`TRUSTED_SERVER__INTEGRATIONS____*`. Configuration values are exposed to your module via +`Settings::integration_config()`. + +```toml +[integrations.my_integration] +endpoint = "https://example.com/api" +timeout_ms = 1000 +rewrite_scripts = true +``` + +### 2. Create the integration module + +Add a module under `crates/common/src/integrations//mod.rs` (see +`crates/common/src/integrations/testlight.rs` for reference) and expose it in +`crates/common/src/integrations/mod.rs`. + +Key pieces: + +```rust +#[derive(Deserialize, Validate)] +struct MyIntegrationConfig { + #[serde(default = "default_enabled")] + enabled: bool, + // … +} + +impl IntegrationConfig for MyIntegrationConfig { + fn is_enabled(&self) -> bool { self.enabled } +} + +pub struct MyIntegration { + config: MyIntegrationConfig, +} + +pub fn build(settings: &Settings) -> Option> { + let config = settings + .integration_config::("my_integration") + .ok() + .flatten()?; + Some(Arc::new(MyIntegration { config })) +} + +// Tests or scaffolding code can seed configs without hand-writing JSON: +settings + .integrations + .insert_config( + "my_integration", + &serde_json::json!({ + "enabled": true, + "endpoint": "https://example.com/api" + }), + )?; +``` + +`Settings::integration_config::` automatically deserializes the raw JSON blob, +runs [`validator`](https://docs.rs/validator/latest/validator/) on the type, and +drops configs whose `is_enabled` returns `false`. Always derive/implement +`Validate` for schema enforcement and implement `IntegrationConfig` (typically +wrapping a `#[serde(default)] enabled` flag) so operators can toggle +integrations without code changes. + +### 3. Return an `IntegrationRegistration` + +Each integration registers itself via a `register` function that returns an +`IntegrationRegistration`. This object describes which HTTP proxies and HTML +rewrites the integration exposes: + +```rust +pub fn register(settings: &Settings) -> Option { + let integration = build(settings)?; + Some( + IntegrationRegistration::builder("my_integration") + .with_proxy(integration.clone()) + .with_attribute_rewriter(integration.clone()) + .with_script_rewriter(integration) + .with_asset("my_integration") + .build(), + ) +} +``` + +Any combination of the three vectors may be populated. Modules that only need +HTML rewrites can skip the `proxies` field altogether, and vice versa. The +registry automatically iterates over the static builder list in +`crates/common/src/integrations/mod.rs`, so adding the new `register` function +is enough to make the integration discoverable. + +### 4. Implement `IntegrationProxy` for endpoints + +Implement the trait from `registry.rs` when your integration needs its own HTTP entrypoint: + +```rust +#[async_trait(?Send)] +impl IntegrationProxy for MyIntegration { + fn routes(&self) -> Vec { + vec![ + IntegrationEndpoint::post("/integrations/my-integration/auction"), + IntegrationEndpoint::get("/integrations/my-integration/status"), + ] + } + + async fn handle( + &self, + settings: &Settings, + req: Request, + ) -> Result> { + // Parse/generate synthetic IDs, forward upstream, and return the response. + } +} +``` + +Routes are matched verbatim in `crates/fastly/src/main.rs`, so stick to stable paths +(`/integrations//…`) and register whichever HTTP methods you need. The shared context +already injects Trusted Server logging, headers, and error handling; the handler only +needs to deserialize the request, call the upstream endpoint, and stamp integration-specific +headers. + +### 5. Implement HTML rewrite hooks (optional) + +If the integration needs to rewrite script/link tags or inject HTML, implement +`IntegrationAttributeRewriter` for attribute mutation and +`IntegrationScriptRewriter` for inline `