Handling Hundreds of Files in Housekeeper

[mikaell]

Handling Hundreds of Files in Housekeeper

ping @dnil, @moonso, @henrikstranneheim, @jemten, @patrikgrenfeldt

The Three-body problem

Chromograph integration is stuck in the intersection of CG, Housekeeper and
Scout. The reason is not knowing how the many (hundreds?) picture files
should be handled and also to generate a configuration file pointing to the
correct files. This is an invitation for discussion. Here are suggested
solutions to the problem.

An ideal solution should:

• keep track of files + delete files
• not create a bloated view in Housekeeper
• be conceptually true to existing software
• sustainable in the long run
• swiftly implemented

Tar-ball (current)

The current solution is to create a tar-ball and hand it to Housekeeper. This solution
is not working at the moment. Problems arise when unpacking it is implossible
to know the filenames and therefore generate a configuration to Scout.

A solution is to guess filesnames with hearuistics and a strict naming convention.

(+)

(-)
Who handles the tracking and deleting of the unpacked files?
Not working at the moment

Do Nothing

Add hundreds of picture files as is into existing software.

(+)
No implementation
Conceptually true to Housekeeper

(-)
The view from housekeeper get will be bloated

Squash Housekeepers View

Tweak Housekeeper's default view by squashing series of filenames -example below. Verbose flag would show everything.

$ housekeeper get soundsnake
[...]   
/home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_[1-22].png 
[...]

$ housekeeper get soundsnake -V
[...]
384502 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_10.png |
384503 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_11.png |
384504 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_12.png |
384505 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_13.png |
384506 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_14.png |
384507 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_15.png |
384508 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_16.png |
384509 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_17.png |
384510 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_18.png |
384511 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_19.png |
384512 | /home/proj/stage/housekeeper-bundles/soundsnake/2020-02-21/soundsnake_gatkcomb_rhocall_vt_af_chromograph_sites_1.png |
[...]

(+)
Conceptually true to Housekeeper
Housekeepers view is not bloated
Small implementation

(-)

Add Directory Handling in Housekeeper

Add functionality to Housekeeper to add and handle directories (adding dirs is already possible, but not delete them).

(+)
Small implementation
Housekeepers view is not bloated

(-)
Changes Housekeeper concept
Have not thought through semantics of this change

Merge PNGs Into One

Instead of handling hundreds of picture files, merge them into one.

(+)
Housekeepers view is not bloated

(-)
Loose granularity and flexibility
Not future proof for adding functionality

Rewrite Chromograph in Javascript

Rewrite picture generation to be done in Scout just in time.

(+)
?

(-)
Unknown implementation time
Unknown problems

---

Other solutions are welcome! Preferably this can be decided quickly.

[mikaell]

I vote for squashing the default view.

[dnil]

I'd say squashing is a good option, regardless of how our three-body problem decomposes? Did you read the SF book by the way, or are you referring to the original physics issue? 😸

[dnil]

Rewrite to js lacks immediacy, and can always be done later without interfering with anything else, as long as the track data is anyway passed to Scout, so leave that one off for now.

Merge is still somewhat tempting to me, but it may be slightly slower to implement, is indeed somewhat more rigid as well as a little less immediate in conveying information to the user.

[moonso]

With regards to #71 we need to skynda långsamt whatever solution is chosen

[mikaell]

Did you read the SF book by the way, or are you referring to the original physics issue? 😸

Both I suppose. Haven't got around reading the book yet. I limit myself to one thick book per year.

[mikaell]

New Tags For Housekeeper

Type	Old Tags	Suggested New Tags
upd (regions)	ACC5322A2, chromograph	ACC5322A2, chromograph, upd, regions
upd (site)	ACC5322A2, chromograph	ACC5322A2, chromograph, upd, sites
Tcov	ACC5322A2, chromograph	ACC5322A2, chromograph, tcov

-replace ACC5322A2 with sample_id.

[jemten]

Do we need a chromosome tag as well or are you planning to get that from the filename later?

[mikaell]

Do we need a chromosome tag as well or are you planning to get that from the filename later?

Ideograms will be placed in public/static and committed into Scout.

[jemten]

Ideograms will be placed in public/static and committed into Scout.

Right! Then, I'll drop MIP's functionality to generate ideograms