az ml environment create fails some file files present in the --build-context

[marrrcin]

Describe the bug

My whole build context has 14MB (which is far less than declared limit of 300MB).
When I include 3 additional files (2x CSV + 1x XLSX) in the build context (which are not actually required by the Dockerfile at all!), the build fails. If those files are excluded - the build proceeds normally.

See the reproduction steps below.

Command Name
az ml environment create

Extension Name: ml. Version: 2.11.0.

Errors:
The command is successful, however the build fails immediately (with no build logs at all) in the Azure ML Studio.

Environment image build status
❌ Failed

To Reproduce:

1. Clone https://github.com/marrrcin/azure-ml-environment-build-issue/tree/master
2. Run az ml environment create --name {} -w {} -g {} --build-context . -d Dockerfile in the root of the cloned repository

Expected Behavior

Build succeeds.

Environment Summary

macOS-12.5-x86_64-i386-64bit, Darwin 21.6.0
Python 3.10.8
Installer: HOMEBREW

azure-cli 2.42.0

Extensions:
storage-preview 0.8.3
ml 2.11.0

Dependencies:
msal 1.20.0
azure-mgmt-resource 21.1.0b1

Additional Context

If I change the .amlignore to exclude the data folder:

!Dockerfile
data

then the build will proceed and finish successfully, which means that something in the data folder breaks the build context in the Azure ML Studio, but those are just plain CSV / XLSX files. I've tested the process and including ANY of them, break the platform with no logs.

I've also tested this by running the same from Linux, same result (failure).

[yonzhan]

route to CXP team

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details

---

Describe the bug

My whole build context has 14MB (which is far less than declared limit of 300MB).
When I include 3 additional files (2x CSV + 1x XLSX) in the build context (which are not actually required by the Dockerfile at all!), the build fails. If those files are excluded - the build proceeds normally.

See the reproduction steps below.

Command Name
az ml environment create

Extension Name: ml. Version: 2.11.0.

Errors:
The command is successful, however the build fails immediately (with no build logs at all) in the Azure ML Studio.

Environment image build status
❌ Failed

To Reproduce:

1. Clone https://github.com/marrrcin/azure-ml-environment-build-issue/tree/master
2. Run az ml environment create --name {} -w {} -g {} --build-context . -d Dockerfile in the root of the cloned repository

Expected Behavior

Build succeeds.

Environment Summary

macOS-12.5-x86_64-i386-64bit, Darwin 21.6.0
Python 3.10.8
Installer: HOMEBREW

azure-cli 2.42.0

Extensions:
storage-preview 0.8.3
ml 2.11.0

Dependencies:
msal 1.20.0
azure-mgmt-resource 21.1.0b1

Additional Context

If I change the .amlignore to exclude the data folder:

!Dockerfile
data

then the build will proceed and finish successfully, which means that something in the data folder breaks the build context in the Azure ML Studio, but those are just plain CSV / XLSX files. I've tested the process and including ANY of them, break the platform with no logs.

I've also tested this by running the same from Linux, same result (failure).

Author:	marrrcin
Assignees:	-
Labels:	question, extension/ml, customer-reported, Machine Learning, Service Attention, Auto-Assign
Milestone:	-

[navba-MSFT]

Adding service team to look into this.

@azureml-github. Could you please look into this once you get a chance ?

[luigiw]

@vizhur can you help to take a look at this?

[vizhur]

Hard to guess without the error, but I would suspect 1MB limit for the Docker context size. The limit is coming from the backend, not Docker itself, due to current limitations in ACR Tasks.