Instructions and installation for inf2 instances are broken

[chkp-assafme]

🐛 Describe the bug

Working by the instructions here: https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md
to create an inf2 instance for running LLama2, the installation breaks.

Error logs

Attempting uninstall: packaging
Found existing installation: packaging 21.3
Uninstalling packaging-21.3:
Successfully uninstalled packaging-21.3
WARNING: Failed to remove contents in a temporary directory '/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/~ackaging-21.3.dist-info'.
You can safely remove it manually.
WARNING: Failed to remove contents in a temporary directory '/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages/~ackaging'.
You can safely remove it manually.
Attempting uninstall: pytest
Found existing installation: pytest 7.4.3
Uninstalling pytest-7.4.3:
Successfully uninstalled pytest-7.4.3
Attempting uninstall: astroid
Found existing installation: astroid 3.0.2
Uninstalling astroid-3.0.2:
Successfully uninstalled astroid-3.0.2
Attempting uninstall: pylint
Found existing installation: pylint 3.0.3
Uninstalling pylint-3.0.3:
Successfully uninstalled pylint-3.0.3
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-api-core 1.34.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<4.0.0dev,>=3.19.5, but you have protobuf 4.25.1 which is incompatible.
neuronx-cc 2.12.54.0+f631c2365 requires protobuf<3.20, but you have protobuf 4.25.1 which is incompatible.
Successfully installed SecretStorage-3.3.3 astroid-2.5 bracex-2.4 captum-0.6.0 cfgv-3.4.0 coloredlogs-15.0.1 coverage-7.3.4 flatbuffers-23.5.26 grpcio-1.60.0 grpcio-tools-1.54.2 html5lib-1.1 humanfriendly-10.0 identify-2.5.33 intel_extension_for_pytorch-2.1.0 jaraco.classes-3.3.0 jeepney-0.8.0 keyring-24.3.0 lazy-object-proxy-1.10.0 markdown-3.5.1 mccabe-0.6.1 mock-5.1.0 more-itertools-10.1.0 mpmath-1.3.0 mypy-1.3.0 mypy-extensions-1.0.0 nh3-0.2.15 ninja-1.11.1.1 nodeenv-1.8.0 onnx-1.14.1 onnxruntime-1.15.0 orjson-3.9.10 packaging-23.2 pkginfo-1.9.6 pre-commit-3.3.2 protobuf-4.25.1 psutil-5.9.5 pygit2-1.13.1 pylint-2.6.0 pynvml-11.5.0 pyspelling-2.8.2 pytest-7.3.1 pytest-cov-4.1.0 pytest-mock-3.12.0 pyyaml-6.0 readme-renderer-42.0 rfc3986-2.0.0 sympy-1.12 toml-0.10.2 torchpippy-0.1.1 twine-4.0.2 wcmatch-8.5 wrapt-1.12.1
WARNING: Ignoring invalid distribution -ackaging (/opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages)
Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com, https://pip.repos.neuron.amazonaws.com
Collecting numpy==1.22.4 (from -r requirements/neuronx.txt (line 2))
Downloading numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.9 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.9/16.9 MB 15.8 MB/s eta 0:00:00
Requirement already satisfied: protobuf==4.25.1 in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from -r requirements/neuronx.txt (line 3)) (4.25.1)
Collecting grpcio-tools==1.48.2 (from -r requirements/neuronx.txt (line 4))
Downloading grpcio_tools-1.48.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 10.9 MB/s eta 0:00:00
Requirement already satisfied: neuronx-cc in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from -r requirements/neuronx.txt (line 5)) (2.12.54.0+f631c2365)
Requirement already satisfied: torch-neuronx in /opt/aws_neuron_venv_pytorch/lib/python3.8/site-packages (from -r requirements/neuronx.txt (line 6)) (1.13.1.1.13.0)
Collecting transformers-neuronx (from -r requirements/neuronx.txt (line 7))
Downloading https://pip.repos.neuron.amazonaws.com/transformers-neuronx/transformers_neuronx-0.9.474-py3-none-any.whl (194 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 194.7/194.7 kB 13.3 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of grpcio-tools to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements/neuronx.txt (line 4) and protobuf==4.25.1 because these package versions have conflicting dependencies.

The conflict is caused by:
The user requested protobuf==4.25.1
grpcio-tools 1.48.2 depends on protobuf<4.0dev and >=3.12.0

To fix this you could try to:

1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Installation instructions

I installed torch by using the python ts_scripts/install_dependencies.py --neuronx --environment=dev command from https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md as well as whatever was already in the AMI.

Model Packaing

Didn't get to the stage of working with model

config.properties

No response

Versions

python serve/ts_scripts/print_env_info.py does not exist

Repro instructions

All I did was try the instructions in https://github.com/pytorch/serve/blob/master/examples/large_models/inferentia2/llama2/Readme.md
as is.

Possible Solution

No response

[chkp-assafme]

I just did a verification on the side. The following two lines create the issue by themselves:
protobuf==4.25.1
grpcio-tools==1.48.2

Both are in requirements/neuronx.txt

This was apparently changed in #2824 two weeks ago (see commit: b8eb8a0)

I tried to remove the limit completely and it installs (although there are many warnings along the way).
Once I did that I managed to get everything working (I did not do a thorough test to see if there are any adverse effects)

[mreso]

Hi @lxning, can you please take a look at this?

[namannandan]

@chkp-assafme This issue has been fixed in this PR: #2887
Follow up PR to simplify the INF2 example further: #2893