Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache results for subject/pipeline #82

Merged
merged 34 commits into from
Oct 26, 2022
Merged

Conversation

gcattan
Copy link
Collaborator

@gcattan gcattan commented Oct 22, 2022

This PR implements #63. It allows caching the result obtained for a subject in a dataset with a particular pipeline.
The PR also provides a Firebase connector to store and share results online. To enable firebase storing, the credentials must be shared privately between users or stored as a secret in the git repository.

I think it is a nice feature for offline analysis because quantum emulation is really taking a long time. We could also inherit from moabb's WithinSessionEvaluation to enable caching there too. This will make also the example on bi2012 easier to reproduce.

@toncho11, @qbarthelemy FYI

@gcattan gcattan marked this pull request as ready for review October 22, 2022 19:02
Copy link
Member

@qbarthelemy qbarthelemy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok for me.
But tests must be corrected.

pyriemann_qiskit/datasets/utils.py Outdated Show resolved Hide resolved
Co-authored-by: Quentin Barthélemy <q.barthelemy@gmail.com>
@gcattan
Copy link
Collaborator Author

gcattan commented Oct 25, 2022

Thank you @qbarthelemy ! The connection test is failing now because I do not have access to this repository secrets. I can either comment the failing test (firebase will not be available within gh actions) or send you the credentials by mail.

@qbarthelemy
Copy link
Member

I've just updated your role, from maintainer to admin.
You should be able to modify gh actions.

@gcattan
Copy link
Collaborator Author

gcattan commented Oct 25, 2022

Thank you :) I am updating the secret and re-running the workflow.

gcattan and others added 2 commits October 25, 2022 12:34
Co-authored-by: Quentin Barthélemy <q.barthelemy@gmail.com>
Co-authored-by: Quentin Barthélemy <q.barthelemy@gmail.com>
@toncho11
Copy link
Collaborator

How is Firebase used? To store results?

test_firebase_connection
@gcattan
Copy link
Collaborator Author

gcattan commented Oct 25, 2022

Yes! It is used to store result, so we can for example split the work between different computers and gather the results on firebase, and use previous results instead of running another analysis. Here is a basic example from the test file:

    for subject in dataset.subjects_:
        X, y = dataset.get_data(subject)
        pipeline.fit(X, y)
        y_pred = pipeline.predict(X)

        cache.add(subject, y, y_pred)
        score = balanced_accuracy_score(y, y_pred)
        scores[subject] = score

    for subject in dataset.subjects_:
        y, y_pred = cache.get_result(subject)
        score = balanced_accuracy_score(y, y_pred)
        assert score == scores[subject]

The connection test to Firebase is making the pipeline failed. The same secret is used on my repository with no problem (gcattan#13). So I think the workflow is running into an untrusted mode because the base branch is from a fork repository (https://docs.github.com/en/actions/security-guides/encrypted-secrets#using-encrypted-secrets-in-a-workflow). This is why I will comment the connection test for the moment.

@gcattan gcattan merged commit e3d4bda into pyRiemann:main Oct 26, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants