Moving the data files and raw processing to a separate repository

[tamasgal]

I wanted to ask your opinion about moving the data files including the raw data processing workflows to a separate repository.
We maintain Corpuscles.jl (https://github.com/JuliaPhysics/Corpuscles.jl) which is a Julia package for accessing the data files provided by Particle and the data update process is manually done.
If we had a separate git repository, we could easily integrate and even automate it.

In KM3NeT we are currently discussing a similar approach for neutrino fluxes and the general acceptance of the idea is pretty good.

[henryiii]

Ahh, sorry, missed this. Though couldn't you integrate this repository, and just ignore the Python parts? They aren't large.

[tamasgal]

All good! Yes sure, that's the approach we have right now. It just feels a bit like building upon an "implementation detail / private api" but I guess the data directory structure and files are quite consolidated now so I am fine with it ;)

Let me close this then.

[henryiii]

As long as we know someone is using it, we'll try to be careful. ;)

[tamasgal]

😜thanks

[eduardo-rodrigues]

Hi @tamasgal! Apologies that I then forgot this issue from you.

I tend to agree that the data files are rather connected to the package, as we digest some and produce others, and would prefer to keep things as is. This being said, I have since ages (more or less literally) a start of work to consolidate a couple of files and simplify a bit (largely what we massage into our CSV). I will try and push forward on that, though time is often against me.

[tamasgal]

No worries Eduardo :) I certainly don't want to trigger more work just because of a single external package. So far we simply manually check and update the data sources and it's not a big deal at all. I will probably go ahead and automate it based on the current structure and whenever there is time or more need for consolidation I'll refactor.