Merge internal - #14
Merged
Merged
Conversation
Improves conversion of control flow, expression-based parameters, and schedule triggers. Co-authored-by: Isaac
* Revert "Update GitHub actions (#10)" This reverts commit f6b98ec. * Improve expression resolution, control flow parsing, and setup (#11) * Refactor expression parsing (#1) * Improve control flow conversion (#2) * Update repo structure * Format modules * Initial commit * Initial release * Improve preparer coverage for managed ingestion pipelines (#2) * Lakeflow connect ingestion pipeline handling * Add required connection parameters * Add marketplace file (#3) --------- Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com> Co-authored-by: service-jira-pub-repo-auto <service.jira-pub-repo-auto@databricks.com>
## Changes Bridge notebooks (generated by the IfCondition, Switch, and ForEach preparers to evaluate complex ADF expressions that can't be inlined into a task's parameters) were missing the standard `# Databricks notebook source` header that every other flowx-generated notebook has, causing `bundle validate` to reject them as "not a notebook" for any pipeline containing one. - Added `render_bridge_notebook()` to `code_generator.py`, consolidating the three previously independent, byte-for-byte-identical private renderers (`if_condition.py`, `switch.py`, `for_each.py`) into one shared function that prepends the notebook header + command separator. - Updated all three call sites (including both call sites in `for_each.py::_resolve_for_each_inputs_with_bridge`) to use the shared renderer, each passing a `title` naming the source activity. - Removed the three now-dead private renderer functions. ### Linked issues Resolves #7 ### Tests - [ ] manually tested - [x] added unit tests - [ ] added integration tests
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR merges public repo changes. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> N/A ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [ ] added unit tests - [ ] added integration tests --------- Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com> Co-authored-by: service-jira-pub-repo-auto <service.jira-pub-repo-auto@databricks.com>
There was a problem hiding this comment.
Pull request overview
This PR consolidates internal changes that (a) ensure generated “bridge” notebooks are emitted as valid Databricks notebooks (with the required # Databricks notebook source header) and (b) simplifies the “agentic” classification model by removing explicit “skill” identifiers from inventory/gap outputs, along with associated documentation and tooling updates.
Changes:
- Add a shared
render_bridge_notebook()renderer that includes the Databricks notebook header, and switch IfCondition/Switch/ForEach bridge notebook generation to use it. - Remove agentic “skill” identifiers from activity classification/inventory/gap/placeholder structures and update tests/docs accordingly.
- Add SDK-based Databricks App deployment notebook/helpers and update docs/CI configuration.
Reviewed changes
Copilot reviewed 37 out of 42 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/test_until_agentic_handler.py | Removes assertions tied to deprecated agentic skill fields. |
| tests/unit/test_preparers.py | Adds unit tests asserting bridge notebooks start with the Databricks notebook header. |
| tests/unit/test_code_generator.py | Adds unit tests for the new shared bridge notebook renderer. |
| tests/unit/test_adf_loader.py | Updates classification/inventory tests to reflect removal of skill mapping. |
| tests/integration/test_end_to_end.py | Updates integration assertions to no longer require agentic skill strings. |
| src/flowx/translator/engine.py | Stops threading agentic skill strings into gaps/placeholders; adjusts placeholder messaging. |
| src/flowx/preparer/workflow_preparer.py | Removes skill-specific hint text from placeholder stub notebooks. |
| src/flowx/preparer/code_generator.py | Introduces render_bridge_notebook() that emits a full notebook (header + separator + body). |
| src/flowx/preparer/activity_preparers/switch.py | Uses the shared bridge notebook renderer and supplies a descriptive header title. |
| src/flowx/preparer/activity_preparers/if_condition.py | Uses the shared bridge notebook renderer and supplies a descriptive header title. |
| src/flowx/preparer/activity_preparers/for_each.py | Uses the shared bridge notebook renderer for inputs-bridge notebooks and supplies a header title. |
| src/flowx/parser/adf_loader.py | Changes AGENTIC_TYPES to a set and simplifies classify_activity() to return only strategy; removes skill from inventory JSON. |
| src/flowx/models/ir.py | Removes agentic_skill / recommended_skill fields from IR models. |
| src/flowx/models/adf_ast.py | Removes agentic_skill from inventory item model. |
| skills/flowx-setup/SKILL.md | Updates Genie Code deployment guidance to prefer the SDK notebook path. |
| skills/flowx-migrate/references/workflow.md | Updates guidance to describe agentic translation as LLM-assisted work from ARM JSON (not “invoke skill X”). |
| skills/flowx-discover/SKILL.md | Updates inventory examples and agentic-handling instructions to remove skill mapping references. |
| skills/flowx-convert/SKILL.md | Updates agentic-gap handling docs/examples to remove plugin-skill routing fields. |
| skills/flowx-convert/references/activity-mapping.md | Updates mapping references to remove external skill identifiers. |
| scripts/bootstrap.sh | Renames temp get-pip file to flowx-branded name. |
| README.md | Reworks project positioning/usage/setup docs (skills + MCP server) and updates phase naming. |
| pyproject.toml | Pins hatchling build-system requirement range. |
| Makefile | Expands uv.lock scrubbing to also rewrite files.pythonhosted URLs and strip size metadata. |
| docs/README.md | Updates docs site repo URL. |
| docs/content/docs/installation.mdx | Major rewrite: separate Genie Code (hosted MCP app) vs local harness installation paths. |
| docs/content/docs/architecture.mdx | Updates repository links in documentation. |
| docs/bun.lock | Renames docs workspace package name. |
| docs/app/layout.config.tsx | Updates GitHub links to the new org repo URL. |
| docs/app/(home)/page.tsx | Updates GitHub link to the new org repo URL. |
| CODEOWNERS | Adds default codeowners team for the repository. |
| app/README.md | Documents the new SDK-based deployment notebook and clarifies when CLI deploy is required. |
| app/deploy_helpers.py | Adds SDK-based helpers for staging and deploying the Databricks App bundle. |
| app/deploy_app.py | Adds a Databricks notebook to deploy the MCP server app via the Databricks SDK (no CLI). |
| .github/workflows/push.yml | Adds a fork/draft gating job and wires ci/fmt behind it. |
| .github/workflows/docs-release.yml | Pins Bun version and moves Pages deploy to a protected runner group. |
| .github/ISSUE_TEMPLATE/feature.yml | Adds a feature request issue form. |
| .github/ISSUE_TEMPLATE/bug.yml | Adds a bug report issue form. |
| .github/dependabot.yml | Adds Dependabot config for uv updates with exclusions. |
| .github/codecov.yml | Adds Codecov coverage status thresholds. |
| .build-constraints.txt | Updates pinned hatchling version/hashes in build constraints. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # PRs from forks are to be tested by the reviewer(s) / maintainer(s) before merging. | ||
| not-a-fork: | ||
| runs-on: ubuntu-latest | ||
| if: github.event_name == 'pull_request' && !github.event.pull_request.draft && !github.event.pull_request.head.repo.fork |
Comment on lines
+89
to
+92
| try: | ||
| workspace_client.workspace.delete(package_workspace_directory, recursive=True) | ||
| except Exception: | ||
| pass # first deploy: nothing to delete |
Comment on lines
+102
to
+107
| try: | ||
| workspace_client.apps.get(name=app_name) | ||
| return False | ||
| except Exception: | ||
| workspace_client.apps.create_and_wait(App(name=app_name, description=app_description)) | ||
| return True |
Comment on lines
+58
to
71
| AGENTIC_TYPES: set[str] = { | ||
| "ExecuteDataFlow", | ||
| "Until", | ||
| "SqlServerStoredProcedure", | ||
| "AzureFunction", | ||
| "WebHook", | ||
| "Custom", | ||
| "ExecuteSSISPackage", | ||
| "AzureMLExecutePipeline", | ||
| "GetMetadata", | ||
| "Validation", | ||
| "Fail", | ||
| "Script", | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR merges fixes missing
# Databricks notebook sourcelines for generated notebooks.Linked issues
N/A
Tests