Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Saving calculations during mosaic map & final layer activations #1

Closed
skochanny opened this issue Oct 21, 2019 · 2 comments
Closed
Labels
enhancement New feature or request

Comments

@skochanny
Copy link
Collaborator

Currently, each mosaic map generation re-calculates final layer activations even if these have been calculated previously. In order to save time, calculated final layer activations should be saved to disk and re-used if the model and tfrecords list are the same.

Proposal: record model & tfrecord list into a JSON file and associate this combination with a filename for the saved final layer activations (CSV). During subsequent calls, this file should be loaded directly instead of re-calculated

@skochanny skochanny added the enhancement New feature or request label Oct 21, 2019
@jamesdolezal
Copy link
Owner

jamesdolezal commented Dec 8, 2019

First implementation in 8003359

Currently, the Mosaic class will save final layer activations to both CSV and PKL (python cache file) after calculation. PKL is much faster to read/write, but cannot be read in a spreadsheet, so both are valuable. The ActivationsVisualizer class can load from either, and will also cache UMAP calculations in PKL format.

The Mosaic class still needs to be able to read from saved CSV and/or PKL format.

@jamesdolezal
Copy link
Owner

Now implemented in b26b5a4. Layer activations are cached to PKL and optionally exported to CSV during calculation, and both can be imported by future models to remove the need for additional calculations.

jamesdolezal pushed a commit that referenced this issue Feb 6, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 6, 2021
jamesdolezal pushed a commit that referenced this issue Feb 6, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 6, 2021
jamesdolezal pushed a commit that referenced this issue Feb 6, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 6, 2021
jamesdolezal pushed a commit that referenced this issue Feb 6, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 6, 2021
jamesdolezal pushed a commit that referenced this issue Feb 6, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 6, 2021
jamesdolezal pushed a commit that referenced this issue Feb 7, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 7, 2021
jamesdolezal pushed a commit that referenced this issue Feb 7, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 7, 2021
jamesdolezal pushed a commit that referenced this issue Feb 7, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 7, 2021
jamesdolezal pushed a commit that referenced this issue Feb 7, 2021
Deleted File

Approved-by: Sara Kochanny <skochanny@uchicago.edu>
Approved-by: James Dolezal <jamesmdolezal@gmail.com>
jamesdolezal added a commit that referenced this issue Feb 7, 2021
@ltzg ltzg mentioned this issue Jun 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants