Added dev mode to install_from_src.py

[msaroufim]

Description

I wanted to add a -e mode to our pip install . scripts so I could make local changes to python files and immediately have them reflected without having to reinstall.

As I was doing that I found a few opportunities for cleanup

1. Single function to install all binaries
2. Removed uninstall scripts and replaced by --force-reinstall
3. Added an environment=dev mode to mimic our install_dependencies.py script
4. Changed the minimum python version to 3.8 to reflect what's said in our documentation
5. Got rid of __pycache__ cleanup, that's what our .gitignore is for

Tests

(serve) ubuntu@ip-172-31-62-14:~/serve$ python ts_scripts/install_from_src.py --environment=dev ------------------------------------------------------------------------------------------ Environment headers ------------------------------------------------------------------------------------------ Torchserve branch: srcclean

torchserve==0.6.0
torch-model-archiver==0.6.0

Python version: 3.9 (64-bit runtime)
Python executable: /home/ubuntu/anaconda3/envs/serve/bin/python

Versions of relevant python libraries:
captum==0.5.0
future==0.18.2
numpy==1.23.2
nvgpu==0.9.0
psutil==5.9.2
pygit2==1.6.1
pylint==2.6.0
pytest==7.1.2
pytest-cov==3.0.0
pytest-mock==3.7.0
requests==2.28.1
torch==1.12.0+cpu
torch-model-archiver==0.6.0b20220912
torch-workflow-archiver==0.2.4b20220912
torchaudio==0.12.0+cpu
torchserve==0.6.0b20220912
torchserve-nightly==0.6.3
torchtext==0.13.0
torchvision==0.13.0+cpu
transformers==4.11.0
wheel==0.37.1
torch==1.12.0+cpu
torchtext==0.13.0
torchvision==0.13.0+cpu
torchaudio==0.12.0+cpu

Java Version:

OS: Ubuntu 18.04.6 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: N/A
CMake version: version 3.22.3

In directory /home/ubuntu/serve | Executing command pip install --force-reinstall -e .

Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Obtaining file:///home/ubuntu/serve
Preparing metadata (setup.py) ... done
Collecting Pillow
Using cached Pillow-9.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
Collecting psutil
Using cached psutil-5.9.2-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB)
Collecting future
Using cached future-0.18.2-py3-none-any.whl
Collecting packaging
Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting wheel
Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Collecting pyparsing!=3.0.5,>=2.0.2
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Installing collected packages: wheel, pyparsing, psutil, Pillow, future, packaging, torchserve
Attempting uninstall: wheel
Found existing installation: wheel 0.37.1
Uninstalling wheel-0.37.1:
Successfully uninstalled wheel-0.37.1
Attempting uninstall: pyparsing
Found existing installation: pyparsing 3.0.9
Uninstalling pyparsing-3.0.9:
Successfully uninstalled pyparsing-3.0.9
Attempting uninstall: psutil
Found existing installation: psutil 5.9.2
Uninstalling psutil-5.9.2:
Successfully uninstalled psutil-5.9.2
Attempting uninstall: Pillow
Found existing installation: Pillow 9.2.0
Uninstalling Pillow-9.2.0:
Successfully uninstalled Pillow-9.2.0
Attempting uninstall: future
Found existing installation: future 0.18.2
Uninstalling future-0.18.2:
Successfully uninstalled future-0.18.2
Attempting uninstall: packaging
Found existing installation: packaging 21.3
Uninstalling packaging-21.3:
Successfully uninstalled packaging-21.3
Attempting uninstall: torchserve
Found existing installation: torchserve 0.6.0b20220912
Uninstalling torchserve-0.6.0b20220912:
Successfully uninstalled torchserve-0.6.0b20220912
Running setup.py develop for torchserve
Successfully installed Pillow-9.2.0 future-0.18.2 packaging-21.3 psutil-5.9.2 pyparsing-3.0.9 torchserve-0.6.0b20220912 wheel-0.37.1

In directory /home/ubuntu/serve | Executing command pip install --force-reinstall -e model-archiver

Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Obtaining file:///home/ubuntu/serve/model-archiver
Preparing metadata (setup.py) ... done
Collecting future
Using cached future-0.18.2-py3-none-any.whl
Collecting enum-compat
Using cached enum_compat-0.0.3-py3-none-any.whl (1.3 kB)
Installing collected packages: enum-compat, future, torch-model-archiver
Attempting uninstall: enum-compat
Found existing installation: enum-compat 0.0.3
Uninstalling enum-compat-0.0.3:
Successfully uninstalled enum-compat-0.0.3
Attempting uninstall: future
Found existing installation: future 0.18.2
Uninstalling future-0.18.2:
Successfully uninstalled future-0.18.2
Attempting uninstall: torch-model-archiver
Found existing installation: torch-model-archiver 0.6.0b20220912
Uninstalling torch-model-archiver-0.6.0b20220912:
Successfully uninstalled torch-model-archiver-0.6.0b20220912
Running setup.py develop for torch-model-archiver
Successfully installed enum-compat-0.0.3 future-0.18.2 torch-model-archiver-0.6.0b20220912

In directory /home/ubuntu/serve | Executing command pip install --force-reinstall -e workflow-archiver

Looking in indexes: https://pypi.org/simple, https://pip.repos.neuron.amazonaws.com
Obtaining file:///home/ubuntu/serve/workflow-archiver
Preparing metadata (setup.py) ... done
Installing collected packages: torch-workflow-archiver
Attempting uninstall: torch-workflow-archiver
Found existing installation: torch-workflow-archiver 0.2.4b20220912
Uninstalling torch-workflow-archiver-0.2.4b20220912:
Successfully uninstalled torch-workflow-archiver-0.2.4b20220912
Running setup.py develop for torch-workflow-archiver
Successfully installed torch-workflow-archiver-0.2.4b20220912
(serve) ubuntu@ip-172-31-62-14:~/serve$ python
Python 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:22:55)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import ts
exit()
(serve) ubuntu@ip-172-31-62-14:~/serve$ git status
On branch srcclean
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   ts_scripts/install_from_src.py
    modified:   ts_scripts/utils.py

no changes added to commit (use "git add" and/or "git commit -a")
(serve) ubuntu@ip-172-31-62-14:/serve$ git add ts_scripts/
(serve) ubuntu@ip-172-31-62-14:/serve$ git commit -m "Add Dev install ^C
(serve) ubuntu@ip-172-31-62-14:/serve$ code CONTRIBUTING.md
(serve) ubuntu@ip-172-31-62-14:/serve$ git status^C
(serve) ubuntu@ip-172-31-62-14:~/serve$ git status
On branch srcclean
Changes to be committed:
(use "git reset HEAD ..." to unstage)

    modified:   ts_scripts/install_from_src.py
    modified:   ts_scripts/utils.py

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   CONTRIBUTING.md

(serve) ubuntu@ip-172-31-62-14:/serve$ git add CONTRIBUTING.md
(serve) ubuntu@ip-172-31-62-14:/serve$ git commit -m "Added dev mode to install_from_src.py"
[srcclean 7e26c04] Added dev mode to install_from_src.py
3 files changed, 37 insertions(+), 76 deletions(-)
rewrite ts_scripts/install_from_src.py (78%)
(serve) ubuntu@ip-172-31-62-14:~/serve$ gh pr create
? Where should we push the 'srcclean' branch? pytorch/serve

Creating pull request for srcclean into master in pytorch/serve

? Title Added dev mode to install_from_src.py
? Choose a template pull_request_template.md
? Body
? What's next? Submit
remote:
remote:
To https://github.com/pytorch/serve.git

• [new branch] HEAD -> srcclean
  Branch 'srcclean' set up to track remote branch 'srcclean' from 'origin'.
  ⣽
  Added dev mode to install_from_src.py #1856
  (serve) ubuntu@ip-172-31-62-14:/serve$
  (serve) ubuntu@ip-172-31-62-14:/serve$ git status
  On branch srcclean
  Your branch is up to date with 'origin/srcclean'.

nothing to commit, working tree clean
(serve) ubuntu@ip-172-31-62-14:/serve$ pre-commit install
pre-commit installed at .git/hooks/pre-commit
(serve) ubuntu@ip-172-31-62-14:/serve$ git status
On branch srcclean
Your branch is up to date with 'origin/srcclean'.

Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git checkout -- ..." to discard changes in working directory)

    modified:   ts_scripts/install_from_src.py

no changes added to commit (use "git add" and/or "git commit -a")
(serve) ubuntu@ip-172-31-62-14:/serve$ git add ts_scripts/install_from_src.py
(serve) ubuntu@ip-172-31-62-14:/serve$ git commit -m "lint"
Check python ast.........................................................Passed
Check builtin type constructor use.......................................Passed
Check for case conflicts.................................................Passed
Check docstring is first.................................................Passed
Check for merge conflicts................................................Passed
Check JSON...........................................(no files to check)Skipped
Check Toml...........................................(no files to check)Skipped
Check Yaml...........................................(no files to check)Skipped
Fix End of Files.........................................................Failed

• hook id: end-of-file-fixer
• exit code: 1
• files were modified by this hook

Fixing ts_scripts/install_from_src.py

Mixed line ending........................................................Passed
Trim Trailing Whitespace.................................................Failed

• hook id: trailing-whitespace
• exit code: 1
• files were modified by this hook

Fixing ts_scripts/install_from_src.py

Check vcs permalinks.....................................................Passed
Check that scripts with shebangs are executable..........................Passed
check for not-real mock methods..........................................Passed
use logger.warning(......................................................Passed
type annotations not comments............................................Passed
pycln....................................................................Failed

• hook id: pycln

• files were modified by this hook

  ts_scripts/install_from_src.py:3:0 'import time' was removed! 🔮
  ts_scripts/install_from_src.py:4:0 'import shutil' was removed! 🔮
  ts_scripts/install_from_src.py 2 imports was removed! 🚀

All done! 💪 😎
2 imports were removed, 1 file was changed.

black....................................................................Failed

• hook id: black
• files were modified by this hook

reformatted ts_scripts/install_from_src.py

All done! ✨ 🍰 ✨
1 file reformatted.

isort....................................................................Failed

• hook id: isort
• files were modified by this hook

Fixing /home/ubuntu/serve/ts_scripts/install_from_src.py

(serve) ubuntu@ip-172-31-62-14:/serve$ gisdasdas^C
(serve) ubuntu@ip-172-31-62-14:/serve$ git add ts_scripts/install_from_src.py
(serve) ubuntu@ip-172-31-62-14:/serve$ git commit -m "lint"
Check python ast.........................................................Passed
Check builtin type constructor use.......................................Passed
Check for case conflicts.................................................Passed
Check docstring is first.................................................Passed
Check for merge conflicts................................................Passed
Check JSON...........................................(no files to check)Skipped
Check Toml...........................................(no files to check)Skipped
Check Yaml...........................................(no files to check)Skipped
Fix End of Files.........................................................Passed
Mixed line ending........................................................Passed
Trim Trailing Whitespace.................................................Passed
Check vcs permalinks.....................................................Passed
Check that scripts with shebangs are executable..........................Passed
check for not-real mock methods..........................................Passed
use logger.warning(......................................................Passed
type annotations not comments............................................Passed
pycln....................................................................Passed
black....................................................................Passed
isort....................................................................Passed
[srcclean 270d300] lint
1 file changed, 12 insertions(+), 8 deletions(-)
(serve) ubuntu@ip-172-31-62-14:/serve$ git p^C
(serve) ubuntu@ip-172-31-62-14:/serve$ git add ts_scripts/install_from_src.py ^C
(serve) ubuntu@ip-172-31-62-14:/serve$ git push
Counting objects: 4, done.
Delta compression using up to 16 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 455 bytes | 455.00 KiB/s, done.
Total 4 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
To https://github.com/pytorch/serve.git
7e26c04..270d300 srcclean -> srcclean
(serve) ubuntu@ip-172-31-62-14:~/serve$ B

[codecov]

Codecov Report

Merging #1856 (7cff32a) into master (9890d0a) will increase coverage by 3.28%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1856      +/-   ##
==========================================
+ Coverage   41.67%   44.95%   +3.28%     
==========================================
  Files          55       63       +8     
  Lines        2282     2609     +327     
  Branches        1       60      +59     
==========================================
+ Hits          951     1173     +222     
- Misses       1331     1436     +105     

Impacted Files	Coverage Δ
model-archiver/model_archiver/model_packaging.py	90.00% <0.00%> (ø)
workflow-archiver/workflow_archiver/version.py	100.00% <0.00%> (ø)
...w-archiver/workflow_archiver/workflow_packaging.py	89.65% <0.00%> (ø)
...hiver/workflow_archiver/workflow_archiver_error.py	100.00% <0.00%> (ø)
...el-archiver/model_archiver/model_archiver_error.py	100.00% <0.00%> (ø)
model-archiver/model_archiver/version.py	100.00% <0.00%> (ø)
...iver/workflow_archiver/workflow_packaging_utils.py	68.68% <0.00%> (ø)
...l-archiver/model_archiver/model_packaging_utils.py	54.96% <0.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[msaroufim]

I may just be seeing things but this PR has actually been reliably succeeding on Windows :O