Test Maintenance

[swahtz]

Overview

ƒVDB has four sets of tests that run on Pull Requests:

1. PyTests
   These tests test functionality on the Python/PyTorch API exposed via the pybind11 bindings. These test a combination of individual pieces of functionality (i.e. grid or jagged tensor creation) as well as full pipelines (i.e. gaussian splatting or rendering).
2. GTests
   These tests are C++ gtests which test public functionality of libfvdb. They are designed as unit tests for individual pieces of pipeline functionality (i.e. gaussian splat projection functions).
3. Documentation Tests
   These tests run on any python code blocks in the markdown of our fvdb/docs directory. This is to ensure any examples we provide in documentation do not break.
4. Continuous Benchmarking Tests
   These tests are meant to help ensure there isn't a performance regression on key operators. These tests measure the runtime of a number of common operators (i.e. convolution) in different configurations or in whole pipelines (i.e. a convolutional Unet). This test records the historical runtimes of these tests into a file fvdb/tests/benchmark/output.json on a branch named gh-pages. This test will fail if the runtime of any test exceeds 2x the runtime of the last run.

Test Data

If test data cannot be generated on-the-fly by a test and needs to be persistently stored in a file, this test data should be stored in an external repository:

https://github.com/voxel-foundation/fvdb-test-data/tree/main/unit_tests

Note: This repo will be moved to the ASWF/openvdb organization on Github in the future for better visibility.

The data in this repository is grouped in directories named after the test files which reference them. If new data required for a test is added to this repository, the git hash for the desired commit should be updated as the GIT_TAG in our cmake infrastructure here:

https://github.com/NVIDIA-Omniverse/openvdb/blob/feature/fvdb/fvdb/src/cmake/get_test_data.cmake

Caching Test Data

We use CPM (CMake Package Manager) to fetch and cache the data from the fvdb-test-data repo. Because we may build and test on different machines in CI, it's important that the same CPM_SOURCE_CACHE settings are used for building and testing. Therefore, the fVDB GHA Workflows set the CPM_SOURCE_CACHE environment variable to the same path for both the build and the gtest task.

If for some reason caching cannot be used, the environment variable must be unset for both build and test.