-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AML Environments are not fully-reproducible #1088
Comments
We are investigating this issue. Will keep you updated. |
what linux distro does AMLCompute use? My guess is there was a distro update. |
Yes that's what we are checking with compute team. BTW, thx for the reproducible example. But I guess below wouldn't ever worked in the linux compute node? |
Our issue is that when we publish a pipeline, we expect it to run in the exact same way on the same environment (including distro). If the code worked for 9 months and we don't change anything, it shouldn't break (without at least a heads up). basically, we had a script arg, However, 9 months ago, we removed So
|
Thx for the details. Now I got it. We just confirmed we have regression on blobfuse update and was able to confirm below code worked with the old version of it: |
Our issue isn't that we can't do this anymore: |
There was a regression from the storage team specifically when paths have a “\” that we had to work with them on rolling back. They are making a fix to handle this scenario, but we are also hardening our system to test this specific use case before releasing. |
We don't care about |
@swanderz with the same environment definition AzureML guarantees reproducibility, unless
For the second case consider to use base_image:@sha256=digest or at least version tag if AzureML base image is used as a base image If client was updated, new version can have new version as a default base image, so if environment got instantiated for the run and not retrieved from workspace ( |
Looked like a drafted a response yesterday and never actually posted it... The derived image wasn't deleted from ACR, you can see that in both run instances the image GUID was the same. My understanding is that I'd love to be wrong though. My thinking right now that pinning |
Sorry, I don't quite understand the problem. You are correct that azure-storage-fuse is not included into the dockerfile/base image. We don't have a python layer on the base images. That gets created from the Encironment object from the conda dependencies. Please provide more details if I misinterpreted your problem |
@vizhur , I appreciate you hanging in here with me. Our problem is that we expected the same environment that we had been using in a Published Pipeline to stay the same. Unfortunately, blobfuse was upgraded and it broke our pipeline. If blobfuse isn’t in the docker image, and it isn’t a pin-able python package, how can we harden our environment to prevent future breaking changes introduced by blobfuse (and perhaps other similarly integrated packages)? |
as I can see from the sample you shared, you did hit the cache as there is no build step. so the environment itself is reproducible. so the image itself is fine, but that was a service side regression as far as I understand |
closing this issue, as I just heard from the support engineer that:
|
SDK version:
1.7.0
region:
West US II
We had the below snippet unchanged in our codebase since at least 9/16/2019. It is a
train.py
script that is used as part of aHyperDriveStep
. Fortunately, we never referenced this'c:\\\\temp\\output'
folder, but, unfortunately, this undocumented change in service behavior has brought all of our pipelines for this workstream (including scheduled production PIpelines to a grinding halt).As of August 3 7:56 AM PST, we could call the beloq snippet in a
ScriptRun
inAMLCompute
without issue. Less than 24 hours later, on Aug 4, 2020 7:30 AM PST, this snippet throws the following error.Here is a notebook that demonstrates reproducible example. I've simplified it to be just a
ScriptRunConfig
, rather than aPIpelineRun
with aHyperDriveStep
working run id:
HD_6b2da7a7-5dfa-4e62-b529-00c802d4307f_4
August 3 7:56 AM PSTfailing run id:
HD_7c714c83-0670-480f-bc64-e7f8c76237c6_3
August 4, Aug 5, 2020 7:30 AMboth runs were in the same experiment in our production workspace, and even used the same pre-built Docker image from ACI,
avaprditsmlsvc8080365804.azurecr.io/azureml/azureml_60fa7a8e306e9ff54f8235b956576310
The text was updated successfully, but these errors were encountered: