What can we do now (if we have the time) to build QGreenland in the open?

[MattF-NSIDC]

See related issue: #693

Currently QGreenland is built on private NSIDC servers. It would be great to migrate to a public (i.e. "cloud") build system (if we have the time) before ending our grant to leave as much of QGreenland in the public sphere as possible.

Are there synergies with ADC we can leverage to achieve this? For example, if we're going to run processing in the cloud for QGreenland-Net, would moving QGreenland onto that system for its build now put us in a better position when we start QGreenland-Net work?

@twilamoon-science it would be good to have a chat about this sooner than later, because the cost may be lower than we think depending on information we get from ADC.

[mbjones]

@MattF-NSIDC what does the build encompass? For our systems at the Arctic Data Center and DataONE, we've been very happy lately with using Github Actions (GHA) for our public CI builds, and they both build our software products, run our tests in an isolaterd environment to be sure developer builds are clean, and create needed images/artifacts for package managers from tagged releases. For public repositories this is free. Would it fit your needs? Here's an example of one of many repositories that uses this system: https://github.com/NCEAS/metadig-engine/actions

Previously we used a public Jenkins instance for CI builds, but have been gradually migrating everything to GHA.

[MattF-NSIDC]

The build encompasses:

• downloading data from providers: some of this is from our own private cache of data that was in high demand, but not in a public archive, so we reached out to scientists to email it to us; we want to get out of this business and into the business of helping scientists make data FAIR and publish it
• processing that data with individualized steps that make the data more FAIR and compatible with our user needs (e.g. must be in EPSG:3413 for performantly displaying many layers together, must be a reasonable file size, etc.)
• building a QGIS project file for making QGIS "layers" out of each data file we processed in the previous step, populating metadata in the QGIS interface, structuring the layers into a hierarchy of groups, etc.
• zipping the whole shebang and putting it somewhere people can access it

I think the 6-hour limit on GHA is going to be an issue for us here. Each of the individual layers could be easily processed within that limitation, but we're not architected for that yet, but we'd like that to change. That may be a great option for QGreenland-Net. The "private archive" is also a problem but surmountable: we can put that data in a bucket somewhere that could be accessed from GHA instead of having it on a private NFS share. And maybe in the next release of QGreenland, we can remove all those "private archive" layers.

That said, I really like the idea of doing the build in GHA, it's been great for everything else.

[mbjones]

Thanks, that's helpful. I think the -Net part of QGreenland-Net is a nod to us trying to enable network-based data access to the QGreenland platform (e.g., the so called OGDC), so that the benefits of the pre-processing/harmonization you are doing would still be realized, but the package would be able to access a much broader set of data resources from networked data repository locations. So maybe the QGIS tool build would have less "data-included" in the future, and more "data-accessible", making the build easier to automate withing the 6-hour limit? Looking forward to this discussion.

[MattF-NSIDC]

the benefits of the pre-processing/harmonization you are doing would still be realized, but the package would be able to access a much broader set of data resources [...] So maybe the QGIS tool build would have less "data-included" in the future, and more "data-accessible", making the build easier to automate withing the 6-hour limit?

Yes, that matches up with what I've been envisioning QGreenland-Net will look like in my own head as well. This will hugely simplify QGreenland to the point that I think we should make it in to a general-purpose library for compiling QGIS projects from community-sourced pre-harmonized online datasets.

I do wonder if GHA has bandwidth or other limits that might still push us to a different compute platform. I doubt it because for the current target size of the Core package, we're looking at only 5GB of data.

Looking forward to this discussion.

Same here, excited to get started!

[twilamoon-science]

Note for awareness on what collaboration/leveraging might be possible with this funded NSF project - https://grass.osgeo.org/news/2023_09_06_nsf_grant_awarded/.