Monthly development updates

[amercader]

Context

As part of a new initiative started by Link Digital, @wardi, @amercader and @smotornyuk will be funded to work on the CKAN project, on what is known as "core work". That is working in the main CKAN repository and the supported extensions to develop new features, reduce tech debt and improve the developer and maintainer experience. You can find more details about it in this blog post:

https://ckan.org/blog/empowering-open-data-and-beyond-major-initiative-to-boost-ckan-development

We aim to be transparent about what we spend our time on, so we will be providing monthly updates in this discussion (we might change the frequency and publication place if necessary). Feel free to add your comments below to ask questions or know more about particular points (please use a thread below each month rather than the main thread)

[amercader]

April 2024

This was our first month working on CKAN maintenance, so we spent quite a lot of time brainstorming and discussing what we could focus on. As you can imagine it was a big list, so we identified 4 key areas that we will focus on during the coming months.

1. CKAN 2.11 release: there are some major features like Table Designer in the review pipeline, but also major refactorings and improvements to make CKAN more secure and performant. We've been making progress in many pending reviews to get as many of them merged as possible
2. Developer and Maintainer experience: these involve improvements and new tools around the main CKAN repo and the Docker images to provide a better experience when working on CKAN, both from a developer point of view (like enabling VSCode remote debugging) and from the maintainers one (like automating backports) Check out this project to follow along other work being done in that front.
3. DCAT v3: CKAN is all about supporting common standards, and a good, robust DCAT support has always been an important goal. An initial plan has been drawn to build support for the new DCAT version, which involves work in the current ckanext-dcat version to align it with modern CKAN development patterns, before implementing DCAT-AP 3 and DCAT-US 3 support. Your feedback is more than welcome in the linked issue, and you can track progress in this project.
4. File management: There is a lot of room for improvement in how files are managed in CKAN. Think better support for large files, reducing CKAN load when accessing data, features like resumable uploads etc. The actual work plan is not yet defined, but @wardi has put down some initial details in 💺🥩🍷 First-class Files #8197, and we will build on top of great work already done by @smotornyuk .

Apart from these main areas we of course worked on regular maintenance tasks like reviewing incoming pull requests and issues, drafting advisories for security issues, answering support requests etc.

[bjornhagstrom]

Great new that you are making DCAT-AP a priority. I checked the progress of that activity and I am thinking about support for national profiles of DCAT-AP (other then teh US that I guess is america?).Several european countries have national profiles, I myself is active in Sweden and know of a few home grown extentions that support DCAT-AP-SE. The activity you name "Scheming support", is that support for national profiles (or support for the possibility to use/define/support other profiles)?

[amercader]

@bjornhagstrom thanks for your comment. The Scheming support pull request includes preliminary work to integrate ckanext-dcat closer with the standard way site maintainers extend and customize metadata schemas with ckanext-scheming. Up until now ckanext-dcat focused on the mapping between CKAN and metadata and the DCAT based serializations that are mandated by the different DCAT-AP profiles, leaving to each site implementer to connect this to how datasets were created and managed. The goal with this work is to make transparent the connection between the fields defined in the CKAN schema and their DCAT representations, so for instance providing a base schema that implements all properties for dcat:Datasets and dcat:Distributions, taking care of things like the form UI for data collection (see e.g. an example below for dcat:contactPoint), translating it to the RDF serialization, loading codelists if necessary etc. There is also work to better support extensions like HVD, GeoDCAT-AP, etc
So to answer your question, yes, we definitely want to make easier to create and maintain national (and other) profiles and integrate the DCAT specific logic with other CKAN functionality.
Hope this makes sense.

[bjornhagstrom]

Thank you, that sounds great!

[amercader]

May 2024

May really flew by really fast but we managed to make a lot of progress on several fronts:

• The biggest pending pull requests for the upcoming 2.11 release have been reviewed, polished and merged. Table Designer, a new Sessions backend, Improved validation of id fields and more. We also started working on integrating with Snyk to incorporate security upgrades more regularly. The new dev-v2.11 branch release is created and soon we'll start the formal release process, for which we'll ask the community for help, stay tuned!
• @wardi has been busy making lots of parts of CKAN faster: startup time, CLI, template snippets and @kowh-ai is working to improve our Docker images to make them more performant, secure and easier to extend
• Lots of movement on the DCAT front as well, initial scheming support is almost ready to go, plus good discussions around support for extensions like HVD and GeoDCAT-AP taking place

[wardi]

Approximate performance improvements:

• cli/start-up time 20% faster
• cli help commands 100% faster
• page rendering up to 20% faster

[amercader]

June 2024

Steady progress on the same fronts as the previous month: CKAN 2.11, Performance and DCAT

• We presented the current initiative and some of the work done so far in this month's CKAN Monthly Live, check it out if you missed it.
• There was a big push on the CKAN 2.11 release front, with work on improving our different Docker images and provide 2.11 images, which was required to start adding support for 2.11 in the most popular extensions. Perhaps you can help test support on one of the pending ones?
• @smotornyuk created some outstanding documentation for the new ckanext-files extension that he has been working on. Make sure to check it out if you are interested in the all the new possibilities that it offers for files management.
• @wardi continues to improve performance and storage, this time at the database level with autogenerated alembic migrations, cleaning up duplicated foreign keys and indexes (autogenerate and sync migrations with models #8239), and removing the groups and package extras tables (smaller DB, faster updates)