Use nox for testing

[justinchuby]

Description

Use nox to run tests. Only linux and macOS is using nox right now.

Additionally:

1. Use ubuntu-latest as the linux image, upgrading from 18.02
2. Add python 3.10 and 3.11
3. Update pipelines to improve how we set environment variables
4. Upload coverage results to codecov
5. Update googletest to the latest commit

Motivation and Context

Unified and repeatable test runs.

Fixes #4791

cc @jcwchen

[justinchuby]

@jcwchen It says I am missing pybind11 (https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36629&view=logs&j=51991ee2-49eb-5ba6-75d8-d9fa44c75e77&t=2ba8a090-2ce7-5eb6-17ea-8e3db6b5fc5d&l=66). Do you have a clue?

also

      /usr/bin/ld: final link failed: Bad value

https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36629&view=logs&j=51991ee2-49eb-5ba6-75d8-d9fa44c75e77&t=2ba8a090-2ce7-5eb6-17ea-8e3db6b5fc5d&l=411

[jcwchen]

@jcwchen It says I am missing pybind11 (https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36629&view=logs&j=51991ee2-49eb-5ba6-75d8-d9fa44c75e77&t=2ba8a090-2ce7-5eb6-17ea-8e3db6b5fc5d&l=66). Do you have a clue?

This is normal. Later on ONNX will build pybind11 from ONNX's submodule.

According the complete error message, it's more like a Protobuf issue. I thought you have removed installing Protobuf from requirements-release.txt. Using a fixed version of Protobuf (3.20.2) should help to resolve this issue.

[jcwchen]

I was wrong. The issue should be export env variables are not shared in different script. Before building ONNX, you will need to manually set export CMAKE_ARGS="-DONNX_USE_PROTOBUF_SHARED_LIBS=ON" again. The error shows you are using a shared Protobuf (comes from apt-get), but the build is using ONNX_USE_PROTOBUF_SHARED_LIBS=OFF".

[justinchuby]

Yeah that's what I figured too. I am trying to see how I can configure the envvars properly

[justinchuby]

@jcwchen pipelines are now working, but its strange that some pbs are not up to date https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36686&view=logs&j=08e05908-426e-595c-5fcc-fd56e5a3c39e&t=74710ad4-2707-5404-bc87-4a4ad8cca814&l=5412

[justinchuby]

@jcwchen pipelines are now working, but its strange that some pbs are not up to date https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36686&view=logs&j=08e05908-426e-595c-5fcc-fd56e5a3c39e&t=74710ad4-2707-5404-bc87-4a4ad8cca814&l=5412

Could it be caused by system image upgrade?

[jcwchen]

@jcwchen pipelines are now working, but its strange that some pbs are not up to date https://dev.azure.com/onnx-pipelines/onnx/_build/results?buildId=36686&view=logs&j=08e05908-426e-595c-5fcc-fd56e5a3c39e&t=74710ad4-2707-5404-bc87-4a4ad8cca814&l=5412

Perhaps also exclude coverage.xml for git diff?

[justinchuby]

Good idea, although I don’t think it’s coverage because untracked files are not going to cause git to return non zero

“Changes not staged for commit:” has some pb files for log sum exp.

[justinchuby]

Is there a way to read / visualize the files?

[jcwchen]

“Changes not staged for commit:” has some pb files for log sum exp.

Those should be excluded by ':!onnx/onnx-data.proto' ':!onnx/onnx-data.proto3' ':!*output_*.pb' ':!*input_*.pb', IIUC. So actually I don't know what happened... Will take a deeper look

[justinchuby]

I see what’s happening. I reverted a condition

[justinchuby]

@jcwchen Done

[jcwchen]

nit: since we are adding two more environments (3.10 + 3.11) in CIs, probably we can remove two existing environments (e.g., one Python 3.7 and one Python 3.8) to keep the number of CI pipelines.

[justinchuby]

Done

[justinchuby]

Linux getting gmake: *** [Makefile:136: all] Error 2 (used to work) I wonder what it was...

[justinchuby]

I think it is

CMake Deprecation Warning at googletest/CMakeLists.txt:49 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

[justinchuby]

I think it is

CMake Deprecation Warning at googletest/CMakeLists.txt:49 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

hmm no. cmake is 3.25.1

[justinchuby]

google test errors. Do we need to update googletest?

[ 50%] Building CXX object googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
In file included from /home/vsts/work/1/s/.setuptools-cmake-build/googletest/src/googletest/googletest/src/gtest-all.cc:42:
/home/vsts/work/1/s/.setuptools-cmake-build/googletest/src/googletest/googletest/src/gtest-death-test.cc: In function ‘bool testing::internal::StackGrowsDown()’:
/home/vsts/work/1/s/.setuptools-cmake-build/googletest/src/googletest/googletest/src/gtest-death-test.cc:1224:24: error: ‘dummy’ may be used uninitialized [-Werror=maybe-uninitialized]
 1224 |   StackLowerThanAddress(&dummy, &result);
      |   ~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~
/home/vsts/work/1/s/.setuptools-cmake-build/googletest/src/googletest/googletest/src/gtest-death-test.cc:1214:13: note: by argument 1 of type ‘const void*’ to ‘void testing::internal::StackLowerThanAddress(const void*, bool*)’ declared here
 1214 | static void StackLowerThanAddress(const void* ptr, bool* result) {
      |             ^~~~~~~~~~~~~~~~~~~~~
/home/vsts/work/1/s/.setuptools-cmake-build/googletest/src/googletest/googletest/src/gtest-death-test.cc:1222:7: note: ‘dummy’ declared here
 1222 |   int dummy;
      |       ^~~~~
cc1plus: all warnings being treated as errors

[justinchuby]

Need >1.11: https://stackoverflow.com/questions/69935158/dummy-may-be-used-uninitialized

[justinchuby]

This is because we are using a newer gcc with ubuntu 20.04

[justinchuby]

gmake[2]: *** No rule to make target 'googletest/src/googletest/googletest/libgtestd.a', needed by 'onnx_gtests'. Stop.

[jcwchen]

I thought previously all pipelines are green? Any new things added causes googletest to fail?

[justinchuby]

@jcwchen Reverted image versions. Now only macos is failing because of the not found protobuf compiler. Any idea how I can fix that?

[jcwchen]

.

[jcwchen]

@jcwchen Reverted image versions. Now only macos is failing because of the not found protobuf compiler. Any idea how I can fix that?

Cross scripts is sometimes a headache in AZP. I guess the export path in the first script for installing Protobuf from source was not applied in the following scripts? If yes, manually export in the this script before installing onnx like

onnx/workflow_scripts/protobuf/build_protobuf_unix.sh

Line 35 in e3bc058

export PATH=$INSTALL_PROTOBUF_PATH/include:$INSTALL_PROTOBUF_PATH/lib:$INSTALL_PROTOBUF_PATH/bin:$PATH

.

[justinchuby]

I didn't realize there was a path exported. thanks!

[justinchuby]

I need to set an env var that is not PATH. Is there another var I can set?

[jcwchen]

I need to set an env var that is not PATH. Is there another var I can set?

Could you please elaborate? I thought you can directly use any variable name you want?