Allow archiving histories and workflows

[hexylena]

My users report having "too many histories" to deal with effectively. I feel their pain, and there are histories that I'm not going to reference very often but would like to retain forever.

I'd like to propose an "archive" for histories, it's another state that histories can transition into.

Properties:

• no modifications allowed (unless unarchived) (this need not be done for the initial implementation)
• not displayed by default (in the context of the "old" history menu where you have to click the advanced -> deleted to see deleted histories)

Update (2016-05-16): I've found the same issue with workflows. There are workflows which I should retain for historical reasons, but will never use again (or are broken, but we want to record exactly what was run)

[blankenberg]

To add to this, I'd actually find just a 'locked' attribute to be great to have separately from archived.

This is particularly important for e.g. published histories (especially ones referenced in papers) -- it is a sad day when you accidentally add some cruft to that perfectly created history that you want to share (instead of the copy you made).

[blankenberg]

Instead of just archived, could we create "Groups" for histories, where "Archived" could be a particular group? That way we can organize things even a bit better.

[hexylena]

@blankenberg sorry, missed your replies. A locked attribute would be amazing, I definitely agree w/r/t published histories.

Groups is an interesting thought. What sort of other groups would you envision having?

[hexylena]

Update: added workflows to this list. Same problem. I want them to be locked down and moved to a different area, out of my sight.

[lparsons]

This sounds like a great idea. I fully support it. I'd like to add another suggestion: the ability to delete underlying datasets in a history (to conserve space), but save the tool. Pretty much the same as a deleted dataset, but without deleting the item in the history. It would be great for users to be able to archive (online and offline) certain outputs (counts, fold changes, plots, etc.) but not some of the redundant or big data (raw fastq stored elsewhere, bam files, etc.). Of course, preserving the provenance of those final outputs would be needed.

Not sure if I made this totally clear, so please let me know if further clarification would be helpful or if there is a better place to record this.

[hexylena]

related: #3269

[bimbam23]

Ideas on how to archive data which is old, unused, ...

• add location where to archive data (maybe in galaxy.yml?)
• find out how old datasets are or if they are still used?
• define live time of data when uploading
• change color (gray) if dataset is archived
• make request of archived data if need

[hexylena]

I think I heard some discussion somewhere that an "archive histories" function could also enable moving them to colder, tape storage (for admins that configure it) which would be super fantastic.

[nomadscientist]

This would be brilliant! I have major issues with accidentally modifying histories I previously shared as examples, I'm a big fan of this. Plus presumably this could make them citable as well. i.e. 'made this tool, history is here for its use, cite me for being awesome' or some such.

[bernt-matthias]

Like the idea:

moving them to colder, tape storage

Wanted something like this for a long time.

[bimbam23]

It's alive again ;-)

[bgruening]

This is the current history table.

id | create_time | update_time | user_id | name | hid_counter | deleted | purged | genome_build | importable | slug | published | importing

We could add a new column locked and include a toggle in the UI to freeze a history. Locked histories are not mutable. The complicated part is a little bit where do we need to add the checkpoints if history is locked? We probably also want to permit changing datatypes in this history, e.g. metadata.

@blankenberg would tags instead of groups work? We have the tagging infrastructure in place. We could think about a tag-centric UI where you see all histories with the same tag in a tree-view?

Could the archived or please-archive feature be a tag with special meaning?

[hexylena]

@foellmelanie mentioned this during her talk, @Slugger70 DM'd me about it as well his users want this. Especially in the context of RO-Crate maybe we can finally do something about this.

Archive the history, and move it to another system with storage (e.g. zenodo)

[Slugger70]

Or other cheap storage like an object store (swift, s3 etc.)

[hexylena]

Yes! Cheap object store, exactly. Could be tape. Could be anything, just a method (especially using the File Sources) letting users archive something and have a URL they can point to that sufficiently describes it, while not taking up space.

[dannon]

Pinging @davelopez who presented some ideas for this at a recent meeting; we're hoping to address this soon.

[davelopez]

Yes, will open a planning issue soon and link it here 👍

[davelopez]

The part for histories is already done in #16003

I'll keep this open for the potential Workflow part but I'll move the milestone.