at long last, all of the pieces are on the table: (a) petabytes of raw files from the HPLT pipeline, (b) 450 or so terabytes of the pool, and 44 + 12 + 15 terabytes of final clean, noisy, and poor data after filtering. of these, (a) resides on LUMI-O (with an interim manifesto in /nird/datalake/NS8112K/public/four/pool/lumio.jsonl), while everything else is already pushed to the NIRD download site.
what remains to be done are analytics reports, organizing checksums and download maps, deciding on how to make the pool and raw data useable, and working with @lukasweymann, @gramirez-prompsit, @akutuzov, and the rest of the HPLT team on writing and polishing the download web page(s).
at long last, all of the pieces are on the table: (a) petabytes of
rawfiles from the HPLT pipeline, (b) 450 or so terabytes of thepool, and 44 + 12 + 15 terabytes of finalclean,noisy, andpoordata after filtering. of these, (a) resides on LUMI-O (with an interim manifesto in/nird/datalake/NS8112K/public/four/pool/lumio.jsonl), while everything else is already pushed to the NIRD download site.what remains to be done are analytics reports, organizing checksums and download maps, deciding on how to make the
poolandrawdata useable, and working with @lukasweymann, @gramirez-prompsit, @akutuzov, and the rest of the HPLT team on writing and polishing the download web page(s).