Example/BI Illiteracy

[gcattan]

Issue #8

This is an example of pipeline evaluation with the worst subjects from the Brain Invaders datasets.
The whole dataset took 3 days to run (but in parallel with the simulation for the financial data PR), and eventually failed due to a MOABB bug.

Below you can find an example with the worst subjects of the VR dataset only (6 "subjects").

A few comments:

• QuantumSVM is in fact a classical SVM here. I disabled it because it returned 50% systematically. @toncho11 do you have time to look into it when you come back? From memory, we obtained something around 0.75 with bi2012. This is weird.
• I think the voting classifier is not appropriate here. We probably want to find the labels on which quantum and classical MDM differ and train an XGBoost on these specific samples.

We can already discuss this example, and proceed to some adjustments before running the whole dataset.

@qbarthelemy FYI

[qbarthelemy]

Is it possible to display all non-quantum classifiers on a side, and all quantum classifiers on the other side of the figure?
With a vertical dotted line between them.

[gcattan]

Ok. It will look like this but with a dotted line instead (result for the 1 subject in the dataset):

There is still some problem with the QSVC, which we need to investigate, but do you see other pipelines to test?

[qbarthelemy]

It's ok for me.

[gcattan]

Super, thanks :) We just need to solve the issue with QSVC. I see it is working correctly with the loans dataset, and I see I have the same problem if I skip the wrapper, and create the pipeline directly with sklearn. Maybe it is just wrongly parametred.

[gcattan]

@toncho11 FYI
I spent some time investigating the bug with QSVM. Here are a few observations:

• The bug also appears when bypassing the QuantumClassifierWithDefaultRiemannianPipeline and using directly the QuanticSVM.
• I played with some values of C and gamma without effect
• I have tried to revert from AerSimulator to BasicAer in quantum_devices.py, without effect.

QSVM is still able to generalize somehow with other datasets, and we have the results from NBT. So, I guess it is a matter of finding which parameters need to be adapted.

[gcattan]

QSVM accuracy is 100% on the training data. Seems overfitted.

[gcattan]

QSVC codebase significantly changed since the last time we ran QSVC on EEG data. I asked for some help in the qiskit-ml repo, and after checking what I could (simulator, quantum kernel, C, and max_iter) I do not see a problem with QSVC (hard to be 100% confident however).
Remains the feature_map which encodes the data into a quantum state.
A few observations:

1. Changing the feature map does help.
2. The probabilities are in the form [ [0.8x, 0.1x], [0.8x, 0.1x]] for all trials. Applying a softmax helps.

[gcattan]

Thanks for the review @qbarthelemy !