Missing modules/packages in Run unit tests

[martin-weber]

It looks as the step Run unit tests in Model CI Pipeline has not yet the packages from ci_dependecies.yml or conda_dependencies.yml applied.
If I get it right, the conda packages are first installed later by step the Publish Azure Machine Learning Pipeline when creating the environment.

What is the best way to have the same modules/packages installed for the unit tests and for the later Environment?

[eedorenko]

The pipeline steps are running in a Docker container (mcr.microsoft.com/mlops/python:latest) which has all packages installed from ci_dependecies.yml

[martin-weber]

Thanks @eedorenko . If I understand that right, I need to create a custom container if I want to add packages in ci_dependencies.yml and have them ready for my unit tests.
Is that right?

So far I did not use a custom container as in conda_dependencies.yml the comment says:

# Conda environment specification. The dependencies defined in this file will
# be automatically provisioned for managed runs. These include runs against
# the localdocker, remotedocker, and cluster compute targets.

Does this mean that the dependencies defined here should be available by the CI unit test?
I added lightgbm in my conda_dpendencies.yml (see below) and in the ci_dependecies.yml (see below) but do not find it in the conda and pip packages (see logs below).

conda_dpendencies.yml

name: mlopsohmw_training_env
dependencies:
  # The python interpreter version.
  # Currently Azure ML Workbench only supports 3.5.2 and later.
  - python=3.7.*
  - pip

  - pip:
    # Base AzureML SDK
    - azureml-sdk==1.3.*

    # Minimum required for the scoring environment. Must match AzureML SDK version.
    # https://docs.microsoft.com/en-us/azure/machine-learning/concept-environments
    - azureml-defaults==1.3.*

    # Training deps
    - numpy==1.18.*
    - pandas==1.0.*
    - scikit-learn==0.22.*
    - lightgbm==2.3.*

    # Scoring deps
    - inference-schema[numpy-support]

    # MLOps with R
    - azure-storage-blob

    # current project
    - azureml-dataprep
    - azureml-monitoring

ci_dependencies.yml

name: mlopspython_ci

dependencies:
  # The python interpreter version.
  - python=3.7.*

  # dependencies with versions aligned with conda_dependencies.yml.
  - numpy=1.18.*
  - pandas=1.0.*
  - scikit-learn=0.22.*
  - lightgbm=2.3.*
  # dependencies for MLOps with R.
  - r=3.6.0
  - r-essentials=3.6.0

  - pip=20.0.*

  - pip:
    # dependencies with versions aligned with conda_dependencies.yml.
    - azureml-sdk==1.3.*
    # Additional pip dependencies for the CI environment.
    - pytest==5.4.*
    - pytest-cov==2.8.*
    - requests==2.23.*
    - python-dotenv==0.12.*
    - flake8==3.7.*
    - flake8_formatter_junit_xml==0.0.*
    - azure-cli==2.3.*
    - lightgbm==2.3.*

I added a step to show the environment at the beginning of .pipelines\code-quality-template.yml

- bash: |   
    conda info -e
    conda list
    pip list
  displayName: 'show environment'

These are the outputs:

It is also confusing why the base conda environment is active.
conda info -e

# conda environments:
#
base                  *  /usr/local
mlopspython_ci           /usr/local/envs/mlopspython_ci

conda list

# packages in environment at /usr/local:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
asn1crypto                1.3.0                    py37_0  
ca-certificates           2020.1.1                      0  
certifi                   2020.4.5.1               py37_0  
cffi                      1.14.0           py37h2e261b9_0  
chardet                   3.0.4                 py37_1003  
conda                     4.8.3                    py37_0  
conda-env                 2.6.0                         1  
conda-package-handling    1.6.0            py37h7b6447c_0  
cryptography              2.8              py37h1ba5d50_0  
idna                      2.9                        py_1  
ld_impl_linux-64          2.33.1               h53a641e_7  
libedit                   3.1.20181209         hc058e9b_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 9.1.0                hdf63c60_0  
libstdcxx-ng              8.2.0                hdf63c60_1  
ncurses                   6.2                  he6710b0_0  
openssl                   1.1.1f               h7b6447c_0  
pip                       20.0.2                   py37_1  
pycosat                   0.6.3            py37h7b6447c_0  
pycparser                 2.20                       py_0  
pyopenssl                 19.1.0                   py37_0  
pysocks                   1.7.1                    py37_0  
python                    3.7.5                h0371630_0  
readline                  7.0                  h7b6447c_5  
requests                  2.23.0                   py37_0  
ruamel_yaml               0.15.87          py37h7b6447c_0  
setuptools                46.1.3                   py37_0  
six                       1.14.0                   py37_0  
sqlite                    3.31.1               h7b6447c_0  
tk                        8.6.8                hbc83047_0  
tqdm                      4.44.1                     py_0  
urllib3                   1.25.8                   py37_0  
wheel                     0.34.2                   py37_0  
xz                        5.2.4                h14c3975_4  
yaml                      0.1.7                had09818_2  
zlib                      1.2.11               h7b6447c_3 

pip list

Package                               Version        
------------------------------------- -------------------
adal                                  1.2.2              
antlr4-python3-runtime                4.8                
applicationinsights                   0.11.9             
argcomplete                           1.11.1             
attrs                                 19.3.0             
azure-batch                           8.0.0              
azure-cli                             2.3.1              
azure-cli-command-modules-nspkg       2.0.3              
azure-cli-core                        2.3.1              
azure-cli-nspkg                       3.0.4              
azure-cli-telemetry                   1.0.4              
azure-common                          1.1.25             
azure-core                            1.4.0              
azure-cosmos                          3.1.2              
azure-datalake-store                  0.0.48             
azure-functions-devops-build          0.0.22             
azure-graphrbac                       0.60.0             
azure-identity                        1.2.0              
azure-keyvault                        1.1.0              
azure-loganalytics                    0.1.0              
azure-mgmt-advisor                    2.0.1              
azure-mgmt-apimanagement              0.1.0              
azure-mgmt-appconfiguration           0.4.0              
azure-mgmt-applicationinsights        0.1.1              
azure-mgmt-authorization              0.52.0             
azure-mgmt-batch                      7.0.0              
azure-mgmt-batchai                    2.0.0              
azure-mgmt-billing                    0.2.0              
azure-mgmt-botservice                 0.2.0              
azure-mgmt-cdn                        4.1.0rc1           
azure-mgmt-cognitiveservices          5.0.0              
azure-mgmt-compute                    12.0.0             
azure-mgmt-consumption                2.0.0              
azure-mgmt-containerinstance          1.5.0              
azure-mgmt-containerregistry          3.0.0rc11          
azure-mgmt-containerservice           9.0.1              
azure-mgmt-cosmosdb                   0.12.0             
azure-mgmt-datalake-analytics         0.2.1              
azure-mgmt-datalake-nspkg             3.0.1              
azure-mgmt-datalake-store             0.5.0              
azure-mgmt-datamigration              0.1.0              
azure-mgmt-deploymentmanager          0.2.0              
azure-mgmt-devtestlabs                2.2.0              
azure-mgmt-dns                        2.1.0              
azure-mgmt-eventgrid                  2.2.0              
azure-mgmt-eventhub                   3.0.0              
azure-mgmt-hdinsight                  1.4.0              
azure-mgmt-imagebuilder               0.2.1              
azure-mgmt-iotcentral                 3.0.0              
azure-mgmt-iothub                     0.11.0             
azure-mgmt-iothubprovisioningservices 0.2.0              
azure-mgmt-keyvault                   2.2.0              
azure-mgmt-kusto                      0.3.0              
azure-mgmt-loganalytics               0.3.0              
azure-mgmt-managedservices            1.0.0              
azure-mgmt-managementgroups           0.2.0              
azure-mgmt-maps                       0.1.0              
azure-mgmt-marketplaceordering        0.2.1              
azure-mgmt-media                      1.1.1              
azure-mgmt-monitor                    0.7.0              
azure-mgmt-msi                        0.2.0              
azure-mgmt-netapp                     0.8.0              
azure-mgmt-network                    10.0.0             
azure-mgmt-nspkg                      3.0.2              
azure-mgmt-policyinsights             0.4.0              
azure-mgmt-privatedns                 0.1.0              
azure-mgmt-rdbms                      2.2.0              
azure-mgmt-recoveryservices           0.4.0              
azure-mgmt-recoveryservicesbackup     0.6.0              
azure-mgmt-redis                      7.0.0rc1           
azure-mgmt-relay                      0.1.0              
azure-mgmt-reservations               0.6.0              
azure-mgmt-resource                   8.0.1              
azure-mgmt-search                     2.1.0              
azure-mgmt-security                   0.1.0              
azure-mgmt-servicebus                 0.6.0              
azure-mgmt-servicefabric              0.4.0              
azure-mgmt-signalr                    0.3.0              
azure-mgmt-sql                        0.17.0             
azure-mgmt-sqlvirtualmachine          0.5.0              
azure-mgmt-storage                    8.0.0              
azure-mgmt-trafficmanager             0.51.0             
azure-mgmt-web                        0.44.0             
azure-multiapi-storage                0.2.4              
azure-nspkg                           3.0.2              
azure-storage-blob                    1.5.0              
azure-storage-common                  1.4.2              
azureml-automl-core                   1.3.0              
azureml-core                          1.3.0              
azureml-dataprep                      1.4.3              
azureml-dataprep-native               14.1.0             
azureml-pipeline                      1.3.0              
azureml-pipeline-core                 1.3.0              
azureml-pipeline-steps                1.3.0              
azureml-sdk                           1.3.0              
azureml-telemetry                     1.3.0              
azureml-train                         1.3.0              
azureml-train-automl-client           1.3.0              
azureml-train-core                    1.3.0              
azureml-train-restclients-hyperdrive  1.3.0              
backcall                              0.1.0              
backports.tempfile                    1.0                
backports.weakref                     1.0.post1          
bcrypt                                3.1.7              
bleach                                3.1.4              
certifi                               2020.4.5.1         
cffi                                  1.14.0             
chardet                               3.0.4              
cloudpickle                           1.3.0              
colorama                              0.4.3              
contextlib2                           0.6.0.post1        
coverage                              5.1                
cryptography                          2.9                
decorator                             4.4.2              
defusedxml                            0.6.0              
distro                                1.5.0              
docker                                4.2.0              
dotnetcore2                           2.1.13             
entrypoints                           0.3                
fabric                                2.5.0              
flake8                                3.7.9              
flake8-formatter-junit-xml            0.0.6              
fusepy                                3.0.1              
humanfriendly                         8.1                
idna                                  2.9                
importlib-metadata                    1.5.0              
invoke                                1.4.1              
ipykernel                             5.1.4              
ipython                               7.13.0             
ipython-genutils                      0.2.0              
isodate                               0.6.0              
javaproperties                        0.5.1              
jedi                                  0.16.0             
jeepney                               0.4.3              
Jinja2                                2.11.1             
jmespath                              0.9.5              
joblib                                0.14.1             
jsmin                                 2.2.2              
jsondiff                              1.2.0              
jsonpickle                            1.4                
jsonschema                            3.2.0              
junit-xml                             1.9                
jupyter-client                        6.1.2              
jupyter-core                          4.6.3              
knack                                 0.7.0rc1           
MarkupSafe                            1.1.1              
mccabe                                0.6.1              
mistune                               0.8.4              
mkl-fft                               1.0.15             
mkl-random                            1.1.0              
mkl-service                           2.3.0              
mock                                  4.0.2              
more-itertools                        8.2.0              
msal                                  1.2.0              
msal-extensions                       0.1.3              
msrest                                0.6.13             
msrestazure                           0.6.3              
nbconvert                             5.6.1              
nbformat                              5.0.4              
ndg-httpsclient                       0.5.1              
notebook                              6.0.3              
numpy                                 1.18.1             
oauthlib                              3.1.0              
packaging                             20.3               
pandas                                1.0.3              
pandocfilters                         1.4.2              
paramiko                              2.7.1              
parso                                 0.6.2              
pathspec                              0.8.0              
pexpect                               4.8.0              
pickleshare                           0.7.5              
pip                                   20.0.2             
pkginfo                               1.5.0.1            
pluggy                                0.13.1             
portalocker                           1.7.0              
prometheus-client                     0.7.1              
prompt-toolkit                        3.0.4              
ptyprocess                            0.6.0              
py                                    1.8.1              
pyasn1                                0.4.8              
pycodestyle                           2.5.0              
pycparser                             2.20               
pyflakes                              2.1.1              
Pygments                              2.6.1              
PyJWT                                 1.7.1              
PyNaCl                                1.3.0              
pyOpenSSL                             19.1.0             
pyparsing                             2.4.7              
pyrsistent                            0.16.0             
pytest                                5.4.1              
pytest-cov                            2.8.1              
python-dateutil                       2.8.1              
python-dotenv                         0.12.0             
pytz                                  2019.1             
PyYAML                                5.3.1              
pyzmq                                 17.1.2             
requests                              2.23.0             
requests-oauthlib                     1.3.0              
ruamel.yaml                           0.15.89            
scikit-learn                          0.22.1             
scipy                                 1.4.1              
scp                                   0.13.2             
SecretStorage                         3.1.2              
Send2Trash                            1.5.0              
setuptools                            46.1.3.post20200330
six                                   1.14.0             
sshtunnel                             0.1.5              
tabulate                              0.8.7              
terminado                             0.8.3              
testpath                              0.4.4              
tornado                               6.0.4              
traitlets                             4.3.3              
urllib3                               1.25.8             
vsts                                  0.1.25             
vsts-cd-manager                       1.0.2              
wcwidth                               0.1.9              
webencodings                          0.5.1              
websocket-client                      0.56.0             
wheel                                 0.30.0             
xmltodict                             0.12.0             
zipp                                  2.2.0             

[eedorenko]

Yes, @martin-weber, you understand that right.

• You need to create a custom container if you want to add packages in ci_dependencies.yml and have them ready for my unit tests.
• Only ci_dependencies.yml dependencies is used in the image. conda_dpendencies.yml is used on AzureML compute
• Did you build/pushed your image and reconfigured the pipeline to use it?
• BTW mcr.microsoft.com/mlops/python:openhack has lightgbm installed

[martin-weber]

Thanks @eedorenko. Ok that solves my issue.

• Will create a custom container. Was not sure if its really needed. Should we add a comment to ci_dependecies.yml that the container needs to be rebuilt?
• Yes, I used mcr.microsoft.com/mlops/python:openhack until I ran into this problem Failure: Deploy to ACI  #265 respectively Update SDK to 1.3.0 #266 and switched then to mcr.microsoft.com/mlops/python:latest that solved the Update SDK to 1.3.0 #266 but lead to the issue described above.