[SHOTS] Allows a list of shots to be specified #1103

josh146 · 2021-02-22T17:30:25Z

Context: Expands the functionality of the QNode, by allowing measuring statistics to be course grained over shots in a single evaluation.

Description of the changes:

Consider

>>> shots_list = [5, 10, 1000]
>>> dev = qml.device("default.qubit", wires=2, analytic=False, shots=shots_list)

When QNodes are executed on this device, a single execution of 1015 shots will be submitted.
However, three sets of measurement statistics will be returned; using the first 5 shots,
second set of 10 shots, and final 1000 shots, separately.

For example:

@qml.qnode(dev)
def circuit(x):
    qml.RX(x, wires=0)
    qml.CNOT(wires=[0, 1])
    return qml.expval(qml.PauliZ(0) @ qml.PauliX(1)), qml.expval(qml.PauliZ(0))

Executing this, we will get an output of size (3, 2):

>>> circuit(0.5)
[[0.33333333 1.        ]
 [0.2        1.        ]
 [0.012      0.868     ]]

The output remains fully differentiable.

Benefits:

More fine-grained control over the shots budget, support for shot-adaptive optimizers
The logic is relatively well-contained, so it should be easy to update this to follow any UI changes to how shots are specified.

Drawbacks: Breaks all plugins that depend on QubitDevice, due to two new keyword arguments in expval, var, and sample.

mariaschuld · 2021-02-23T06:44:34Z

Nice!

One thought, we cannot assume that other devices inheriting from Device support sequences of shots. What will happen if this is attempted in those devices?

codecov · 2021-02-23T11:01:07Z

Codecov Report

Merging #1103 (97ed04d) into master (5095918) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1103   +/-   ##
=======================================
  Coverage   97.73%   97.74%           
=======================================
  Files         154      154           
  Lines       11683    11726   +43     
=======================================
+ Hits        11418    11461   +43     
  Misses        265      265

Impacted Files	Coverage Δ
pennylane/_device.py	`96.46% <100.00%> (+0.28%)`	⬆️
pennylane/_qubit_device.py	`98.90% <100.00%> (+0.11%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5095918...97ed04d. Read the comment docs.

pennylane/_device.py

pennylane/_qubit_device.py

co9olguy · 2021-02-23T13:26:27Z

Is analytic=False still needed in the example?

josh146 · 2021-02-23T13:27:40Z

Is analytic=False still needed in the example?

@co9olguy yes, until #1079 is merged, which removes the analytic keyword.

I branched off master rather than #1079 because the logic here is somewhat self-contained, and the tests are not yet passing in #1079.

josh146 · 2021-02-23T15:32:21Z

pennylane/_qubit_device.py

+            # count the basis state occurrences, and construct the probability vector
+            for b, idx in enumerate(indices):
+                basis_states, counts = np.unique(idx, return_counts=True)
+                prob[basis_states, b] = counts / bin_size


I wonder if this can be vectorized, rather than a for loop? Perhaps it's also possible to do away with the if/else, and have a single block of logic for any bin size.

I'm not certain it can easily be vectorized just because the output shape of np.unique is different each loop.

josh146 · 2021-02-23T15:33:53Z

One thought, we cannot assume that other devices inheriting from Device support sequences of shots. What will happen if this is attempted in those devices?

For devices that don't know about self._shot_vector, nothing changes - they will continue to just use the total number of shots. Don't know if this makes sense or not - we could add something to the capabilities dictionary?

chaserileyroberts

I'm not feeling super keen on this right now. It feels like we're adding a ton of complexity for what amounts to a for loop in user code.

chaserileyroberts · 2021-02-23T15:41:23Z

pennylane/_device.py

+    number of shots and the shot vector.
+
+    Args:
+        shot_list (Sequence[int, tuple[int]]): sequence of non-negative shot integers


Why is tuple[int] included in sequence?

At the back of my mind, I wanted to allow (advanced) users and other parts of the codebase to pass shots=[(1, 1e9)], rather than having to do shots=[1] * 1000000000, which will create a large list in memory for absolutely no reason

pennylane/_qubit_device.py

pennylane/_device.py

pennylane/_qubit_device.py

chaserileyroberts · 2021-02-23T15:58:14Z

pennylane/_qubit_device.py

+            # count the basis state occurrences, and construct the probability vector
+            for b, idx in enumerate(indices):
+                basis_states, counts = np.unique(idx, return_counts=True)
+                prob[basis_states, b] = counts / bin_size


I'm not certain it can easily be vectorized just because the output shape of np.unique is different each loop.

chaserileyroberts · 2021-02-23T16:15:54Z

pennylane/_device.py

+            total_shots = shot_list[0] * len(shot_list)
+        else:
+            # Iterate through the shots, and group consecutive identical shots
+            split_at_repeated = np.split(shot_list, np.where(np.diff(shot_list) != 0)[0] + 1)


I'm having a hard time parsing this line.

So the first np.diff(shot_list) != 0 is a boolean array with the first indicies where the shot value changes marked as True, and np.where(...) returns tuple of an array of the indices of these first values. Is that correct?

Yep, essentially. This line splits the shots list, grouping together consecutive and identical elements:

pennylane/_device.py

chaserileyroberts · 2021-02-23T16:18:12Z

pennylane/_device.py

+        if len(set(shot_list)) == 1:
+            # All shots are identical; represent the shot vector
+            # in a sparse format.
+            shot_vector = [(shot_list[0], len(shot_list))]
+            total_shots = shot_list[0] * len(shot_list)


This chunk isn't needed as the code below should correctly handle this case

True, I added this after for two reasons:

I think shots=[n] * N is probably the most common use-case

In the case of shots=[n] * N, this is faster than the second branch.

But yes, the second branch should handle both cases

pennylane/_device.py

Co-authored-by: Chase Roberts <chase@xanadu.ai>

josh146 · 2021-02-23T16:33:07Z

I'm not feeling super keen on this right now. It feels like we're adding a ton of complexity for what amounts to a for loop in user code.

Unfortunately, a for loop in the user code (while giving the same results) will generate separate jobs that may be held up in a remote queue. The idea here is to 'batch' shots on a single QNode eval.

Shot adaptive quantum optimizers typically need to generate many 1-shot expectation values, which is currently not possible to implement in PennyLane without submitting many single-shot jobs.

Note also that the loop is not over single shots, but batches of shots which are vectorized. I imagine arguments of the form shots=[1] * 10000 along with expectation values to be the dominant use case, in which the internal for loop in device.execute will only involve one iteration.

josh146 · 2021-02-23T16:57:48Z

@Thenerdstation have standardized the format to a namedtuple, and made every element a tuple - it cleans it up a lot, and helps remove a lot of redundant lines.

chaserileyroberts

This feels a lot cleaner thank you!

Approval for code structure and readability, but I leave it to nathan and maria to decide if we want to break the existing plugins for this.

josh146 · 2021-02-23T17:23:15Z

This feels a lot cleaner thank you!

Np!

Approval for code structure and readability, but I leave it to nathan and maria to decide if we want to break the existing plugins for this.

I expect #1079 to pre-break all plugins, so took a little liberty here 😆

chaserileyroberts · 2021-02-24T02:30:28Z

I expect #1079 to pre-break all plugins, so took a little liberty here

That's fair

pennylane/_device.py

mariaschuld · 2021-02-24T09:21:48Z

pennylane/_qubit_device.py

-            expectation values of observables
+        shots (int, list[int]): Number of circuit evaluations/random samples used to estimate
+            expectation values of observables. If a list of integers is passed, the circuit
+            evaluations are batched over the list of shots.


I think this will need some more explanation? For example, what is the rule for circuit outputs? If the device is analytic and qml.probs(), qml.sample() is returned, will only the second return value be batched? But will both be batched if the device is non-analytic?

I know that this will somewhat change by shots refactor II, but good to be precise?

Yes, good point. At the moment, anything that relies on samples will get batched if the shots are batched. So that includes samples always, and expval/var/probs in non-analytic mode.

This will definitely be more intuitive after II, because the presence of batched shots will always correspond to batched output.

pennylane/_qubit_device.py

mariaschuld · 2021-02-24T09:28:12Z

tests/test_qubit_device.py

+            qml.RX(x, wires=0)
+            qml.RY(y, wires=0)
+            qml.CNOT(wires=[0, 1])
+            return qml.probs(wires=0), qml.probs(wires=1)


I am actually not sure now, but do we allow for mixed-measurement-process outputs like probs(), expval()? If so, isn't that an important edge case?

This is undefined behaviour at the moment 🤔 This results in a ragged array, and I purposely didn't try to solve it due to an upcoming change in NumPy that will make it really hard to work with object arrays

mariaschuld

Nice one @josh!

Just a few questions before approving, but it looks good to me.

mariaschuld · 2021-02-24T09:30:08Z

tests/test_qubit_device.py

+            qml.device("default.qubit", wires=2, analytic=False, shots=0.5)
+
+        with pytest.raises(ValueError, match="Unknown shot sequence"):
+            qml.device("default.qubit", wires=2, analytic=False, shots=["a", "b", "c"])


Do we need a test for the per-qnode-call way of setting shots?

Allow a list of shots to be specified on a device

564e9d1

josh146 requested a review from mariaschuld February 22, 2021 17:30

josh146 added 3 commits February 23, 2021 15:35

faster implementation

a8a8517

fix

972e4d0

changelog

d660f9e

more

d9dc6d9

josh146 marked this pull request as ready for review February 23, 2021 12:01

josh146 changed the title ~~[WIP] [SHOTS] Allow a list of shots to be specified on a device~~ [SHOTS] Allow a list of shots to be specified on a device Feb 23, 2021

mariaschuld reviewed Feb 23, 2021

View reviewed changes

pennylane/_device.py Outdated Show resolved Hide resolved

mariaschuld reviewed Feb 23, 2021

View reviewed changes

pennylane/_qubit_device.py Show resolved Hide resolved

suggested changes

705c33d

suggested changes

03accba

josh146 commented Feb 23, 2021

View reviewed changes

josh146 requested a review from chaserileyroberts February 23, 2021 15:32

josh146 added the review-ready 👌 PRs which are ready for review by someone from the core team. label Feb 23, 2021

josh146 changed the title ~~[SHOTS] Allow a list of shots to be specified on a device~~ [SHOTS] Allows a list of shots to be specified Feb 23, 2021

chaserileyroberts reviewed Feb 23, 2021

View reviewed changes

Update pennylane/_device.py

90271fd

Co-authored-by: Chase Roberts <chase@xanadu.ai>

josh146 added 4 commits February 24, 2021 00:33

Merge branch 'master' into list-shots

d582f55

more

b3a5f6e

suggested changes

6c87f77

add test

39f690a

docstrings

9a65cc2

chaserileyroberts approved these changes Feb 23, 2021

View reviewed changes

update comments

eaa9542