Added code-coverage.yml using codecov

[msaroufim]

Description

This PR introduces code coverage using codecov https://about.codecov.io/

The code coverage results will show up as a github action showing the total coverage and how much coverage was added or lost in a PR. The action will show up as failing if coverage goes down in a PR. For now we've included

• Tests to ts/
• Test to workflow_archiver and model_archiver

Note that postman does not do code coverage, postman is a way to test APIs not code

Also deleted install_utils shell script since it's not used anywhere and has some remnant pytest related code that may be confusing later on

(base) ➜  serve git:(msaroufim-patch-13) ✗ grep -r "install_utils" .                      
(base) ➜  serve git:(msaroufim-patch-13) ✗

Fixes #1636

Next steps in future PRs

• Handler unit tests will do with this PR [WIP]: Add and Fix handler Unit tests in CI #1672
• Regression test will need to change 1 line and add regression test to CI so it runs on PRs as well

  serve/ts_scripts/regression_utils.py

  Line 56 in a4d5090

  cmd = "python -m pytest -v ./"

• Java coverage - in a separate PR will leverage Jacoco report
• Documentation changes - docs/code_coverage.md will no longer be needed. Will delete when Integrate dead link checker in linting CI #1668 is landed since that README also has broken link linting
• Coverage is around 45% :( - so will need to have some separate PRs to get it closer to 90% on an ongoing basis

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Feature/Issue validation/testing

• CI (see below)
• Codecov bot (see below)
• Coverage showing up here https://app.codecov.io/gh/pytorch/serve
• Add YAML https://docs.codecov.com/docs/common-recipe-list

Checklist:

• Did you have fun?

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (master@46076f1). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 1debc0a differs from pull request most recent head 5b146e7. Consider uploading reports for the commit 5b146e7 to get more accurate results

@@            Coverage Diff            @@
##             master    #1665   +/-   ##
=========================================
  Coverage          ?   45.36%           
=========================================
  Files             ?       64           
  Lines             ?     2590           
  Branches          ?       60           
=========================================
  Hits              ?     1175           
  Misses            ?     1415           
  Partials          ?        0           

Help us with your feedback. Take ten seconds to tell us how you rate us.

[msaroufim]

For example here is the coverage for ts/metrics

[msaroufim]

Checks will look like this

[msaroufim]

This is what the UI will look like to identify specific lines that were not covered

[msaroufim]

Hi @maaquib @lxning @HamidShojanazeri this PR is ready to review now

[lxning]

Could you elaborate the problems of the existing code coverage report? Where do you set up code coverage threshold in CI job?

[msaroufim]

Thanks for the feedback @lxning

Could you elaborate the problems of the existing code coverage report?

1. Generates an HTML report which isn't easy to render on remote machines unless you download the files
2. Is not integrated into CI in an easy-to-use way with visible outputs for anyone to browse
3. Is a point in time measurement and doesn't show you trends with coverage increasing or decreasing or provide aggregations to identify which folders need the most work to expand coverage
4. Does not show relative improvements or decreases in coverage for a PR relative to master
5. Not integrated with a centralized dashboard solution
6. No easy way to aggregate all coverage reports into one large report

codevov solves all of the above, codecov is the default code coverage tool all pytorch projects use because the UX is fantastic. I would highly recommend you click on https://app.codecov.io/gh/pytorch/serve and browse the images above to get a feel for what makes it so nice

Where do you set up code coverage threshold in CI job?

This is easy to do with codecov.yaml https://docs.codecov.com/docs/codecov-yaml, I'll add a 45% threshold for now which we can revisit and increase over time after #1667 and #1670 are landed

[lxning]

@msaroufim I think both ci-gpu and ci-cpu build should have code coverage. lint was already in the existing ci build.

[msaroufim]

@msaroufim I think both ci-gpu and ci-cpu build should have code coverage.

Done

lint was already in the existing ci build.

It's not ideal to have linting in the CI pipeline where tests run since that takes ~20-30min to run whereas our new linting github action takes about 11s. The new black linting pipeline is also tightly integrated with pre-commit where you can setup a config files for all the linting rules you want with a great local UI. https://github.com/pytorch/serve/blob/msaroufim-patch-13/.pre-commit-config.yaml - it's also what torchx and torchdata use and it's a very popular way of doing linting in open source

[mreso]

LGTM

[msaroufim]

@maaquib @lxning I believe we can go ahead and merge this change. As far as integration tests don't do, postman does not do code coverage since it only tests an API. The files in test/pytest are the next obvious candidate to improve code coverage once I also integrate them into Github Action