-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable use of Azure ML Environments and code upload #14
Comments
Could you elaborate how your approach works? Are all of the features of Kedro preserved? |
The main change is to reference the Environment by name:
instead of kedro-azureml/kedro_azureml/generator.py Lines 98 to 100 in b15758c
and to add I'm fairly new to Kedro, so could you clarify which Kedro features you think might be lost? |
We initially force docker-based executions to prevent issues with missing dependencies and code changes. In our MLOps lifecycle, we run valid training jobs always from CI/CD, that's the only way we can be 100% sure that experiments are reproducible - enabling ad-hoc executions with code upload might interfere with that. Having that in mind, if that's the only change that needs to be done and the pipeline works the same, we can adopt that approach as an optional feature. There should be some additional CLI command introduced to give the users easy way to create this Azure environment and then - the config should have a new parameter that switches the execution mode. In the
|
Thanks for your comments and being open to the suggestion!
I understand that CI/CD runs are preferred, but for experimentation this creates a large overhead.
Azure ML runs as I suggest should also be fully reproducible, since they contain both the code and the Environment which was used in the experiment. I do agree that it should be made clear that the code is uploaded and not taken from the Docker image.
By creating an AML Environment based on the
I agree with the need for such a command, this was the next thing on my list. Regarding how to create it, I would also like to support the |
Registered one.
Sounds good to me, go ahead - I'm looking forward to your PR! |
@tomasvanpottelbergh how is your PR going? |
Sorry, I was on vacation, so didn't make much progress. I am first sorting out a few other things to make the plugin work for our workflow, after which I can clean things up and open the PR. Feel free to start a PR already if you were also working on this. |
We're not actively working on it at the moment, so please proceed at your own pace 🙂 |
I am starting to clean up my code to be able to open a PR for this. While doing that, I had some more thoughts I'd like your opinion on:
Looking forward to your thoughts on this! |
It would be great if you show us the flow you've prepared in a form of steps necessary to run Kedro on Azure with the code upload and we can iterate on this from the high level perspective. |
Understood, but currently it's already a two-step process. My fear with automating the environment creation is that any change to the config would need to be reflected in the environment as well. Users might expect this to happen automatically, whereas it would be impractical to update the environment on every run.
I think keeping backward-compatibility would complicate the design. In theory we could keep the
I can also just open a PR to make things more concrete and we could iterate there. |
Is this an expensive process? Maybe it can be done in 2 phases - check for changes and then update only if there are changes.
Go ahead! :) |
In most cases it is, possibly involving file uploads and a Docker build, which needs a compute cluster to run. Checking for changes would be technically difficult to impossible for my workflow to create an environment based on a build context.
Thanks, I will try to have something ready this week. |
Thanks for creating this plugin!
When evaluating it for usage in our projects, I was a bit surprised by the approach to use a Docker image with the code baked in, instead of the Azure ML Environments (or Docker images) with code upload on job submission.
The current approach works fine for production deployments, but it doesn't support the experimentation workflow very well. I modified the plugin locally to support submitting jobs using Environments and code snapshotting. Would this change be welcome as a PR? Or would you like to keep this current approach or have both options?
The text was updated successfully, but these errors were encountered: