Merge v3 into main #1680
Merged
Merge v3 into main #1680
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Remove non-bigquery warehouses * Cleaup some dependency usage, make build * Re-add materialized views * Fix some tests
…re and api's SQL adapters (#1554) * More clearly differentiate core and api's SQL adapters * Remove helper namespace * Fix some opinionated imports and class usage * Fix more import orders
* Consolidate CLI package by moving API and VM directories under it * Fix import order
Merge main to v3
Merge main to main_v3
* Remove gen index * Fix semver * Simplify error message
* Add base dir structure for reading workflow settings, some cleanup * Add a basic workflow_settings reader * Progress with loading YAML via require * Working YAML reader for workflow settings * Tidying * Cleanup bits * Fix relative path for requiring workflow settings * Fix import orders and lint errors * Stop exporting getWorkflowSettings * Move TSLint comment locations * Add explicit include for new compilation_sql BUILD file to cli/api * Fix compilation sql visibility
* Add workflow settings validation * Tidy * Remove return from recursive field check * Fix object lint error * Remove braking run cache option from CLI * Remove console log
* Remove deprecated run cache and gcloud project ID fields * Remove run cache test * Tidying
…CLI (#1585) * Move concurrency and action retries from project config proto to the CLI * Fix typing * Fix tests * Fix optional
Merge main to main_v3 3
* Populate warehouse as BigQuery by default, update tests * Remove log * Add tslint ignore for no string literal
* Update to apache 2 license * Cleanup other MIT licenses
* Stop using ProtobufJS's verify method because it doesn't really do anything * Fix proto validation and tests * Remove buganizer reference * Fix lint
* Add basic actions.yaml and notebook support * Add test for notebook * Add action details extraction from filename, facilitate requiring notebooks, make tests work * Tidy * Fix lint errors * fix * George comments
* Update tests for custom variables to new structure, enforce it for workflow_settings.yaml too * Tidy * Fix tslint
* Move tasks.ts to CLI, and minor cleanup * Fix imports * Fix API test * Fix import order * Add missing build rule
…Dataform Core (#1603) * Contextable config test * Add a test for variable merging, and applying in SQLX config * Add override for default schema * Tidy * Update core tests to use the var from the project config request for auto assertion database
* fix file namings * Add licenses to the CLI too * Tidy usage * Fix build and bump version * Fix core version in test
…to the action config file (#1636) * Update file name to be read as absolute * Fix tests
* Add more tests, throw better errors for invalid declaration action configs * Make file compiler loaders not functions * Tidy * Move notebook cell stripping and compile json to main * Tidy
…ests (#1644) * Add a notebook example project * Fix tests and node_modules usage * Tidy escapers * Fix lint * Tidy * Update backtick escaper to use SQLX tree instead * Tidy * Add newlines
* Make the CLI use and default to workflow_settings.yaml * Tidy * Undo debugging examples change
* Update Dataform CLI npm installs to best-effort * Tidy logic * Prevent errors when package.json already present * Fix lint
Merge to v3 4
* Remove dead resolve code * Tidy * Make returned string empty rather than unresolved
* Make main_test more concise by making filenames optional * Tidy, fix walk * readability comments * Apply simplify to new tests
…ls (#1655) * Remove more dead code, including navigator column descriptors and tools * Remove dead navigator test * Tidy
* Remove warehouse option from the CLI, more tidying * Fix default db location requirement * Make default location and database positional for init * Fix import order * Remove some unused CLI commands, tidy * Remove unused finally block
* v3-js-revert-as-v2 * Fix early return, and tests
…#1656) * Update the v3 proto to use a separate compiled graph, with UX review changes * Tidy * Nest bigquery options, tidy action descriptors * Re-add config protos temporarily, to prevent test breakage * Remove columnar level tags, and compiled graph dataform core version * Move action configs outside the compiled graph * Tidy action fields further * Improve action config namespacing * Add notebook config to compiled graph format compilation * More proto tweaks * Progress applying proto updates within file * Fix core main tests * Fix remaining compilation and testing issues
* Core path tidying * tidy
…est style (#1665) * Migrate core sqlx syntax and assertions context functions tests to _test style * Tidy
* Avoid surpressing CLI errors with inaccurate core not found message * Tidy * Add test for error message * Tidy * Tidy * Make error message less ambigiuous * Fix import order
* Replace cli spec tests with _test format * Fix lint
* Improve generated workflow_settings.yaml field order * nit
* Fix config targets * Fix lint * Add more tests, fix fully qualified dependencies for notebooks
* Remove JS context from SQL files * Tidy * Update example projects * Tidy * Simpler sql escaping and example
* Throw error when calling install if core version is managed by a workflow_settings.yaml * Make the compile method stateless * Make installation happen in a temporary directory, remove package-lock usage * Fix lint * Make installation happen in the temporary directory, add test for weird files * Fix lint * Fix tests * Fix tests
BenBirt
approved these changes
Feb 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merge v3 into main - and keep main as the default tracking branch. Lewis pointed out that our vulnerability scanning, and other systems expect the main branch.
Main v2 is at https://github.com/dataform-co/dataform/tree/main_v2.
After this PR is submitted I will delete the main_v3 branch.
Will merge with a merge commit, in order to preserve history.