Main causes of test failures found

[shuheng-liu]

Causes of Test Failure

To understand the causes of massive test failures,

1. PyTorch tensors (on the low level) are strictly-typed and does not support dynamic casting. For example, one can not add a Double tensor with a Float tensor without explicitly casting one to the other's type.

2. PyTorch supports globally setting the default type of tensors via torch.set_default_dtype(...), which will be used for all tensors subsequently created, until the current process exits. If no calls to torch.set_default_dtype are made, PyTorch will work out default tensor type using some unknown strategies.

3. When python evaluates a script (for import or any other purposes), it runs everything in that module verbatim. In other words, Python is not intelligently only evaluate bar when a user executes from foo import bar. Instead, it runs (evaluates) the entire file foo.py and registers foo.bar to the current namespace.

4. [To be confirmed] When pytest is executed with no arguments, it searches for all files matching the test_*.py pattern, executes them one at a time (in the same process), registers all target functions beginning with test in all files, and runs all registered targets. Note that in this manner, if default types are specified multiple times in different .py files, only the last call takes effect.

5. In our case, default tensor types are different among test files. Therefore, running any individual test files yields no error (which is the common workflow when using an IDE like PyCharm) while running all tests together in a single process results in failure (which is the case for our travis-ci build config).

Behavior

To verify our theory, consider the following ways of runnings tests

1. Scenario A: Running all test files together using

pytest tests/test_*.py 
# or, without arguments
pytest

causes some test cases to fail because of Float vs Double type mismatch

2. Scenario B: Running each test file one at a time using

for file in tests/test_*.py; do
    pytest $file
done

yields no error (except for the technical debt in test_pde_spherical.py)

Solution

Modify the following line in .travis.yml

  - pytest --cov-report term --cov=neurodiffeq/

to do pytest one file at a time.

I'm not familiar with Travis config's syntax, but probably something like

  - pytest tests/test_function_basis.py
  - pytest tests/test_ode.py
  - pytest tests/test_pde.py
  - pytest tests/test_pde_spherical.py
  - pytest tests/test_temporal.py

Is there a more elegant way of doing this?

[shuheng-liu]

Somehow travis ci doesn't recognize the wildcard *, so I couldn't do this:

for file in neurodiffeq/tests/test_*.py; do
    pytest $file
done