[Cheatsheet] Pip requirements examples

[aminsaied]

Provide examples on what is possible for pip installation in azureml environments. (ado work item)

More details:
This ask came from Deepspeed team as they develop their Deepspeed curated environment.

How can we include "non-standard" pip commands in environments? For example:

apex install builds a bunch of kernels (cuda/cpp) and needs to be installed like this:
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
​
which is easy to define in a traditional dockerfile but doing it in the CE setup is a bit tricky

The Conda file's pip section is effectively a pip requirements.txt file and supports most pip options. See doc at:
https://pip.pypa.io/en/stable/reference/pip_install/#requirements-file-format

That includes global-option, for example, but it is not a complete set of pip flags.

If there's one we need here that's not covered there, the only path today is to add it directly into the Dockerfile.

Medium-term, we're working on changes that will store our environments as a Dockerfile + context, which will make advanced use cases like this more straightforward.