SonicTriton tests with Singularity

[kpedro88]

PR description:

SonicTriton tests can now run on either Singularity or Docker. This makes the test setup more accessible (since Docker typically requires superuser permission). An augmented Docker image with PyTorch libraries included is now hosted on DockerHub at https://hub.docker.com/repository/docker/fastml/triton-torchgeo, while the Singularity version of that image is automatically generated and hosted on /cvmfs/unpacked.cern.ch thanks to https://gitlab.cern.ch/unpacked/sync.

A unified script triton is introduced to handle the server for both the Docker and Singularity cases. It also handles using GPU instead of CPU, verbosity, waiting for the server to actually start, and other details. The documentation is updated accordingly.

(This PR also fixes some alarmingly fast link rot in the tests... hopefully Nvidia is done renaming the triton inference server repository.)

PR validation:

Tested on several different machines, and had some other users test as well.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31616/18681

• This PR adds an extra 20KB to repository

[cmsbuild]

A new Pull Request was created by @kpedro88 (Kevin Pedro) for master.

It involves the following packages:

HeterogeneousCore/SonicTriton

@makortel, @cmsbuild, @fwyzard can you please review it and eventually sign? Thanks.
@makortel, @rovere this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[kpedro88]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-09-29-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/9649/console Started: 2020/09/29 20:17

[cmsbuild]

+1
Tested at: 576f4d7
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fb2074/9649/summary.html
CMSSW: CMSSW_11_2_X_2020-09-29-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-fb2074/9649/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2542225
• DQMHistoTests: Total failures: 13
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2542190
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 34 files compared)
• Checked 149 log files, 22 edm output root files, 35 DQM output files

[kpedro88]

@makortel @fwyzard any comments?

[makortel]

The fetch_model.sh is not run by PR or IB tests, right?

[kpedro88]

No, not currently. (At some point, we'll try to introduce a unit test using the local CPU server, but this is not ready at present.)

[makortel]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1