Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make pyro.generic.testing usable as a todo list #2053

Closed
1 of 7 tasks
fritzo opened this issue Oct 3, 2019 · 1 comment
Closed
1 of 7 tasks

Make pyro.generic.testing usable as a todo list #2053

fritzo opened this issue Oct 3, 2019 · 1 comment
Labels
Milestone

Comments

@fritzo
Copy link
Member

fritzo commented Oct 3, 2019

Ideally I'd like to implement a very narrow testing interface with two methods that do not mention pytest and do not require testing code to mention pyro.generic. Their use is equivalent to the following:

# in my_pyro_implementation/test/test_generic.py
import pytest
from pyro.generic.testing import get_test_cases, check_test_case
# n.b. do not import any pyro stuff, including pyro.generic

@pytest.mark.parametrize('test_case', get_test_cases())
def test_generic(test_case):
    try:
        check_test_case(test_case)
    except NotImplementedError as e:
        pytest.xfail(f"Not implemented:\n{e}")

I don't care what these are named or whether they are lists or functions or whatever, but I do think we should separate concerns such that:

  1. all pyro.generic interface is hidden inside these test helpers, and
  2. all testing infra (pytest) is applied outside these test helpers.

I see these helpers as kind of an autoconf mechanism for pyro implementations: running them automatically shows what features are available in a given backend.

(see original discussion in #2049 (comment))

Tasks

  • Raise NotImplementedError on missing backend functionality #2055 raise NotImplementedError in pyro.generic if a backend module is missing an attribute
  • refactor pyro.generic.testing to free users from mentioning pyro
  • add SVI examples to pyro.generic.testing
  • exercise these tests with the pyro backend
  • exercise these tests with the minipyro backend
  • exercise these tests with the numpyro backend
  • exercise these tests with the funsor backend
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant