feat: template connector stage 1#9321
Closed
royendo wants to merge 67 commits into
Closed
Conversation
Adds a new GenerateTemplate RPC endpoint that serves as the single source of truth for YAML format generation. Replaces frontend imperative YAML builders with a centralized backend handler. - Add EnvVarName field to PropertySpec for custom env var name mapping - Populate EnvVarName values across 14 driver specs (S3, GCS, Azure, ClickHouse, Postgres, BigQuery, Snowflake, Redshift, MotherDuck, Athena, MySQL, Druid, Pinot, Salesforce) - Define GenerateTemplate proto messages with validation annotations - Implement handler: auth check, driver validation, property validation, DuckDB rewrite for object/file stores and sqlite - Add comprehensive YAML renderers for connectors and models with precise formatting via yaml.Node tree construction - Implement env var conflict resolution with _1, _2 suffix appending - Add 54 passing test cases covering all drivers, error scenarios, and edge cases Frontend migration to use this RPC will follow as a separate step. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace client-side YAML builders with backend RPC for single source of truth. - New helper: `generateTemplate()` + `mergeEnvVars()` in generate-template.ts - Updated submission paths: submitAddConnectorForm, submitAddSourceForm, saveConnectorAnyway - Updated preview: computeYamlPreview now calls RPC (async, 150ms debounced) - HTTPS connector path preserved (backend doesn't support headers array yet) - Removed dead code: compileSourceYAML, prepareSourceFormData, maybeRewriteToDuckDb - Backend now handles DuckDB rewrite and env var naming (conflict resolution) - Tests: added mergeEnvVars (9 tests) and normalizeConnectorError (11 tests) - Net: 234 lines removed (~27% reduction in submission/preview code) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ector headers
Add HTTPS headers support to the backend GenerateTemplate RPC, enabling removal
of all client-side YAML generation code. The HTTPS connector now uses the same
RPC path as all other connectors.
Backend changes:
- Add `headers` property to HTTPS driver ConfigProperties
- Implement header rendering with sensitive value extraction to .env
- Support Authorization header scheme prefixes (Bearer, Basic, Token, Bot)
- Use `connector.{driver}.{key}` naming convention for header env vars
- Add 4 comprehensive test cases for header handling
Frontend changes:
- Convert key-value UI arrays to maps before RPC submission
- Remove HTTPS-specific YAML preview and submission code
- All connectors now use GenerateTemplate RPC exclusively
- Remove 13 dead functions (compileConnectorYAML, updateDotEnvWithSecrets, etc)
- Consolidate env var naming to backend-only responsibility
Schema cleanup:
- Remove 30 dead `x-env-var-name` annotations (frontend no longer reads them)
- Single source of truth for env var names: backend PropertySpec.EnvVarName
Results:
- -1,317 lines of dead code removed
- +221 lines of backend implementation
- 2,055 frontend tests pass, 62 backend tests pass (4 new HTTPS tests)
The mapping from property Key to env var name doesn't follow a mechanical pattern. Well-known names are shared across drivers (AWS_ACCESS_KEY_ID), some add infixes (AZURE_STORAGE_*), and others diverge entirely (GCS key_id -> GCP_ACCESS_KEY_ID). Each driver must specify explicitly. Also remove personal docs from .gitignore.
- Fix gofmt alignment in connectors.go PropertySpec struct - Remove unused connectorName param from renderDuckDBModelYAML - Use const instead of let for newEnvBlob and connectorYamlBlob
…o — Core types: Template, File, ProcessedProp registry.go — Registry with //go:embed loading of YAML definitions, List(), Get(), ListByTags(), LookupByDriver() render.go — Rendering engine: pre-processes properties (secret extraction, empty filtering, DuckDB derivation), renders Go text/template with [[ ]] delimiters funcmap.go — Template functions: renderProps, indent, quote env.go — Extracted: ResolveEnvVarName, ReadEnvKeys headers.go — Extracted: IsSensitiveHeaderKey, SplitAuthSchemePrefix, HeaderKeyToEnvSegment, ResolveHeaderEnvVarName (with fixed per-call regex compilation bug) duckdb.go — Extracted: BuildDuckDBQuery, matchesExt (with fixed false positive bug for paths like parquet-archive/readme.txt) 32 template definitions (runtime/templates/definitions/) 17 connector templates: s3, gcs, azure, https, postgres, bigquery, snowflake, mysql, athena, redshift, salesforce, clickhouse, duckdb, motherduck, druid, pinot, starrocks 7 DuckDB model templates: s3-duckdb, gcs-duckdb, azure-duckdb, https-duckdb, local-file-duckdb, sqlite-duckdb 8 warehouse model templates: clickhouse-model, postgres-model, mysql-model, bigquery-model, snowflake-model, athena-model, redshift-model, salesforce-model 1 new template: iceberg-duckdb (the motivating use case) Proto definitions Template, TemplateFile, ListTemplatesRequest/Response, GenerateFileRequest/Response, GeneratedFile messages ListTemplates and GenerateFile RPCs on RuntimeService Frontend client auto-generated via Orval Server handlers (runtime/server/templates.go) ListTemplates — returns templates filtered by tags GenerateFile — renders template, optionally writes files + merges .env Tests (4 test files, 32+ test cases) Registry: loading, lookup, tag filtering, sorted output, all definitions valid Render: S3 connector, S3-DuckDB model, Snowflake warehouse model, Redshift no-dev, Iceberg-DuckDB, env var conflicts, empty filtering, output filtering, local file, SQLite Env: explicit name, fallback, conflict, double conflict DuckDB: query building for all formats, false positive fix Headers: sensitive detection, auth scheme splitting, env segment naming Bug fixes from PRD containsExt false positive fixed (basename suffix check) headerKeyToEnvSegment regex compiled at package level Render errors propagated (not silently empty)
…nnector, rewire front end to new templates
…s to put files into local_files in tmp/
Cascading changes from the orval→connectRPC merge: replace `instanceId` / `runtime` store usage with `RuntimeClient` from `useRuntimeClient()` in generate-template, AddDataFormManager, connector-schemas, and submitAddDataForm. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolves conflicts: keeps template-driven approach from branch, regenerates proto/runtime clients for GenerateFile API, updates generate-template.ts to use v2 client. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These were dropped during merge conflict resolution but are still imported by AddDataModal.svelte. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The auto-merge kept the branch's slimmer version, but saveAiConnector.ts (from main) depends on compileConnectorYAML, updateDotEnvWithSecrets, and other functions that were only on main's side. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Missing exports (getSchemaSecretKeys, getSchemaStringKeys, etc.) needed by saveAiConnector.ts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add AI_CONNECTORS and updateRillYAMLWithAiConnector imports that were dropped during merge conflict resolution. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Stage 1 ships the template architecture for connectors that already exist on main. ClickHouse-as-OLAP support (and the new kafka/hudi/mongo source templates that target it) moves to a follow-up. - Drop runtime/templates/definitions/clickhouse-models/* and olap/clickhouse.json - Drop ApacheHudi*, ApacheKafka*, MongoDB* connector icons and their mappings - Pare CH-specific cases from render_test.go and generate_template_test.go; swap remaining CH usages to postgres for env-conflict and validation tests - Add populateSchemaCache test seam to connector-schemas.ts so existing vitest specs (FormValidation, AddDataFormManager, add-source-visibility, connector-schemas) run in isolation Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
To simplify PR: #8981 into sections instead one large PR
Checklist: