How can I use this?

[cdeil]

Hi, I just now saw astrocats and https://astrocats.space/ . This is awesome!

I'm currently considering starting an open TeV gamma-ray source catalog (see gammapy/gamma-sky#33) with the main motivation to have something usable for http://gamma-sky.net/ , but also because I'm frustrated by the fact that there is no open TeV source catalog at the moment.

The TeV catalog would be small (~150 sources, ~20 fields, initially populated from http://www.asdc.asi.it/tgevcat/) and then consist of frequent manual updates.

How can I use this?

Is there an option of having it listed on https://astrocats.space/ and having the catalog repo under https://github.com/astrocatalogs ?
Or would it be better if I do the repo under my own org and / or webpage on my own website (e.g. as a sub-page of gamma-sky.net)?

[lzkelley]

Hi @cdeil, thanks!

A TeV-catalog sounds like it would be a fantastic addition! And I think our framework could be easily adapted for your purposes. We've started a wiki page describing how to create a new catalog --- a process which I'm working on myself as-well. It's only a rough outline at the moment, but will continue to be expanded.

The current procedure is to clone the astrocats source code, in addition to an existing catalog (e.g. supernovae) which is then copied and modified to create a new one. Currently it would be much easier to both contain the repositories under github/com/astrocatalogs and the backend on @guillochon 's server space --- we're happy to host it for you.

[guillochon]

@cdeil So as you guys already have your own domain, it would definitely be possible to have the catalog on your domain and hosted by us (or even hosted by you on your own server). Right now I see what you guys have is a nice map as your home page, but it might be nice to have the catalog and the map be separate pages that are both available through the gamma-sky.net domain.

The first step is what @lzkelley suggested: To get the import script working such that you can use the scripts that generate the web-based catalog. We are happy to host the final product in whichever form you prefer, but it's probably worth discussing what's best before we make a decision. I'll also gladly link to it from the astrocats.space domain once it is at least in a "beta" state.

Feel free to join us in our slack channel: https://astrochats.slack.com/messages/general/

[cdeil]

@lzkelley and @guillochon - Thanks for the quick and positive reply.

My preference would be if the TeV catalog is maintained under your Github repo and domain, to have this more clearly be a community thing that's separate from the Github org I've started https://github.com/gammapy/ and the webpage @vorugantia and I have started http://gamma-sky.net/ .

For years there has been http://tevcat.uchicago.edu/ as a separate catalog and website, and as I mentioned in the other issue, they don't want to make their catalog fullly open. My hope would be that their TeVCat and this new TeV source catalog I want to start here will unite into a community-maintained catalog like what you've set up for other domains of astronomy. For now and in any case, we'd just follow your community open catalog model and do it for TeV sources.

I'd like to get started on this tomorrow morning and see if I can ingest some initial TeV catalog and get it working. Thanks for the link to your wiki and writing up the steps.

Should I start with repos under the gammapy org for now or are does it seem better to you to make repos for this under your Github org now?

[cdeil]

It should just be "Open TeV catalog" and repo name e.g. open-tevcat (to distinguish it from the existing http://tevcat.uchicago.edu/

[cdeil]

Looks like https://astrochats.slack.com/messages/general/ is password-protected. Is anyone with a Slack account able to join or do you have to invite me before I can join? My email is listed on my Github profile page.

[guillochon]

Hi @cdeil, I just invited you.

[guillochon]

@cdeil As for where the repos should live, you should just fork them for now and when you think it's close to being ready we'll fork it into astrocatalogs. I can tell you more over Slack.

[cdeil]

FYI: I'm starting this now here: https://github.com/gammapy/gamma-cat

For the next few days I'll just roll my own Python scripts. I think that's easier for me to get something done, and once it's there you'll see what TeV catalog we have and can better advice if / how using astrocats makes sense.

I did try to follow
https://github.com/astrocatalogs/astrocats/wiki/Catalog-Creation-Guide
but then

git clone git@github.com:astrocatalogs/supernovae.git astrocats/supernovae

and

python -m astrocats supernovae import --tasks internal

did take a lot of space (6 GB I think) and time, and it wasn't even clear to me what the result / output would be.

So if you have the time, I think it would be much better to have the "getting started" guide use an example catalog with very few data (1 MB and few files) that can be processed within seconds, so that people have something useful running quickly and see what astrocats does and try to understand how it works.

[guillochon]

@cdeil, this isn't in that guide I don't think but did you try this:

python -m astrocats catalog import

That runs the example import, which is a much more lightweight set of tasks.

Sorry things aren't working out immediately! We'll try better to make the install process more straightforward.

[lzkelley]

gamma-cat is continuing outside of the astrocats framework.

[cdeil]

Yes, for gamma-cat, I've now decided to write my own Python scripts, that generate a static webpage via Sphinx (coming soon).

I could never quite figure out if the astrocats framework would work for our application, and what benefits and drawbacks it would bring, compared to just rolling our own.

But @lzkelley and @guillochon - please know that even if we're not using astrocats, I'm very grateful for your feedback and help and to show the way how something like this can be set up (text files, git, webpage, community effort, paper). Thank you!

[guillochon]

No hard feelings whatsoever @cdeil! Please keep us posted when gamma-cat becomes available, would be interested to see if any of the choices you made could be inform our future design decisions.

[cdeil]

One choice I've made is that input files are YAML for hierarchical data, and ECSV for tables (see input folder in our repo. For input validation I'm using JSON schema and Python scripts, with the JSON schema again read from YAML files. Basically because I don't like editing JSON files, YAML files are more pleasant.

For the output, the plan (to be implemented in the next days) is to expose the data via index JSON files that list and link to all available data. And for the individual data files I'm not sure yet, I was thinking JSON for hierarchical data, and ECSV for tables. For the summary catalog, I'm planning to offer a FITS and ECSV version. Basically I don't know yet what format users prefer for the output, because we only have very few users so far. We're also developing a Python package to access and work with the data with domain-specific classes (e.g. spectral model, spectral points, morphology model, lightcurve) here and we're using gamma-cat as one of the source catalogs on http://gamma-sky.net/ (not live yet, a preliminary version should go live in the next days) to make browsing the catalog and comparing the MWL data easy. For gamma-sky.net we have a proceeding published, for gamma-cat nothing yet.

So it's different in detail from your choices, but the overall concept is very similar.
I think in the end, we had a lot of domain-specific things, so that using your framework would not have been the right choice. But I might be wrong and should just have taken some time to learn what you have and see if / how our stuff fits in.