Adds CLI and Python invocations for the device test suite #733
Conversation
josh146
added
the
review-ready 👌
Codecov Report
@@ Coverage Diff @@
## master #733 +/- ##
=======================================
Coverage 95.50% 95.50%
=======================================
Files 110 110
Lines 6858 6858
=======================================
Hits 6550 6550
Misses 308 308
Continue to review full report at Codecov.
|
pennylane/plugins/tests/__init__.py
Outdated
|
|
||
| .. code-block:: console | ||
|
|
||
| pl-device-test --device default.qubit --shots 1234 --analytic False |
There was a problem hiding this comment.
Took me a while to figure out that one needs to pip install -e . to get this to work...
Strangely, 2 tests for default.qubit fail? Not essential here, but just wondering.
FAILED pennylane/plugins/tests/test_measurements.py::TestSample::test_sample_values_hermitian[device_kwargs0]
FAILED pennylane/plugins/tests/test_measurements.py::TestSample::test_sample_values_hermitian_multi_qubit[device_kwargs0]
There was a problem hiding this comment.
Took me a while to figure out that one needs to pip install -e . to get this to work...
Yep! The setup.py file needs to be run for the CLI script to be placed on the path.
Strangely, 2 tests for default.qubit fail? Not essential here, but just wondering.
These two tests fail stochastically for me; looks like a tolerance issue. Increasing shots to 10000 fixes it :)
Co-authored-by: Maria Schuld <mariaschuld@gmail.com>
…to device-test-cli
| >>> from pennylane.plugins.tests import test_device | ||
| >>> test_device("default.qubit") |
There was a problem hiding this comment.
Worked great when ran from an ipython session or as part of a Python script! 🙂 💯
In a Jupyter notebook the output was directed into the terminal where jupyter notebook was invoked (and there was no output in the notebook itself).
There was a problem hiding this comment.
Good catch about Jupyter notebook! Do you think this is a common use-case we should support? I think we should not support running tests from within a Jupyter notebook
There was a problem hiding this comment.
Kind of on the fence here 🤔 I think it's not too crucial, though if interactive sessions are supported then a Jupyter notebook would also be an important case. Having said that I think it's not a major point
| @@ -209,7 +251,7 @@ def pytest_runtest_makereport(item, call): | |||
| # and those using not implemented features | |||
| if ( | |||
| call.excinfo.type == qml.DeviceError | |||
| and "not supported on device" in str(call.excinfo.value) | |||
| and "supported" in str(call.excinfo.value) | |||
There was a problem hiding this comment.
How come this needed a change?
There was a problem hiding this comment.
I noticed that not all of the plugins were standardized in their error messages for unsupported behaviour :( For example, the Cirq plugin raises an 'unsupported' error if you attempt to use certain gates it some situations.
There was a problem hiding this comment.
Oh, I see! Hmm, would it be worth adding an extra condition here with or?
| 'package_data': {'pennylane': ['plugins/tests/pytest.ini']}, | ||
| 'include_package_data': True |
There was a problem hiding this comment.
If not for this addition, would plugins/tests/pytest.ini be ignored when the test suite is invoked by using the cli?
There was a problem hiding this comment.
The issue is that the pytest.ini file isn't included in the packaged wheel without this being included! So this forces it to be packaged in the wheel.
By default setuptools will only package up .py files.
There was a problem hiding this comment.
Looks good to me! Very nice one @josh146 😊 Awesome that the test suite will be usable for external devices too
Context:
Description of the Change:
Adds a function for finding the location of the installed tests:
get_device_tests()Adds a function for invoking the tests from within Python:
Adds a convenient CLI script for invoking the tests, that comes installed with PennyLane:
$ pl-device-test --device default.qubit --shots 1234 --analytic False --tb=short -xMakes the plugin test suite documentation visible.
Small modification to allow passing arbitrary device keyword arguments, using
--device-kwargs key1=val1 key2=val2. This flag must come last, as it consumes everything to the right of it.Benefits:
Tests can be easily invoked on external devices even if PennyLane is just
pipinstalled. This will be quite useful for plugins in external repos.Re-uses the existing pytest argument parser in
conftest.pyto avoid code duplicationPossible Drawbacks:
Related GitHub Issues: n/a