Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: datafactory data-flow update does not work #7062

Merged

Conversation

JessicaLHartog
Copy link
Contributor

az datafactory data-flow update is broken and cannot be used. There is a bug where data_flow_data["properties"]["typeProperties"] is not initialized to an empty dictionary and so every time the command is run, the following will appear at the end of the error trace:

in datafactory_data_flow_update
    data_flow_data["properties"]["typeProperties"]["sinks"] = properties[
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
KeyError: 'typeProperties'

To fix this issue (and prevent it from happening again) I've performed some refactoring to:

I based the expected behavior on the CLI documentation - specifically with respect to the following:

typeProperties: (optional) If none provided, a generic and empty definition is created. Including the definitions is best done by copying the JSON from within the Data Factory Studio in the Azure Portal.

This checklist is used to make sure that common guidelines for a pull request are followed.

Related command

az datafactory data-flow create
az datafactory data-flow update

General Guidelines

  • Have you run azdev style <YOUR_EXT> locally? (pip install azdev required)
  • Have you run python scripts/ci/test_index.py -q locally? (pip install wheel==0.30.0 required)

For new extensions:

About Extension Publish

There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update src/index.json automatically.
You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify src/index.json.

Copy link

azure-client-tools-bot-prd bot commented Dec 5, 2023

️✔️Azure CLI Extensions Breaking Change Test
️✔️Non Breaking Changes

Copy link

Hi @JessicaLHartog,
Please write the description of changes which can be perceived by customers into HISTORY.rst.
If you want to release a new extension version, please update the version in setup.py as well.

@microsoft-github-policy-service microsoft-github-policy-service bot added the customer-reported Issues that are reported by GitHub users external to the Azure organization. label Dec 5, 2023
@yonzhan
Copy link
Collaborator

yonzhan commented Dec 5, 2023

Thank you for your contribution! We will review the pull request and get back to you soon.

@JessicaLHartog
Copy link
Contributor Author

@microsoft-github-policy-service agree company="Delphix"

@JessicaLHartog JessicaLHartog force-pushed the fix-datafactory-data-flow-update branch 4 times, most recently from becc1dc to be6c9d5 Compare December 8, 2023 19:13
@JessicaLHartog JessicaLHartog force-pushed the fix-datafactory-data-flow-update branch 2 times, most recently from f8068ec to 4fd99ba Compare January 22, 2024 19:46
@JessicaLHartog
Copy link
Contributor Author

@evelyn-ys Any timeline where this can get looked at? The az datafactory data-flow update command does not work at all, so this fix is pretty important.

@evelyn-ys evelyn-ys merged commit 9101c64 into Azure:main Jan 25, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants