Architecture documentation
This wiki provides a structured overview of the system architecture behind data.norge.no — a national data catalog designed to support the sharing and discovery of public sector data in Norway — as well as adjacent or supporting systems.
The architecture consists of the following subsystems:
- 📝 Registration – For registering and maintaining metadata.
- 🔄 Harvesting – For ingesting and enriching metadata.
- 🌐 Portal – For searching and browsing metadata.
- 📏 Metadata Quality – For computing scores on metadata about datasets.
- 🧑💼 IAM (Identity & Access Management) – For authentication and authorization.
💡 Tip: Click on the images to get a full view of the diagrams. In the full view, every container (deployable unit) is also clickable and will redirect you to the actual repository on GitHub.
Registration is a system for registering and maintaining metadata, intended for organizations that do not have their own systems or data catalog. Employees within these organizations use Registration to create and manage metadata for datasets, data services, general services, and concepts.
The resulting data catalogs are treated like any other external catalog and are harvested through the same harvesting process.
Harvesting is a system for ingesting metadata from data catalogs. These catalogs can either be external, from organizations that maintain their own catalogs, or internal, from catalogs created using the Registration system.
Harvesting supports various resource types, including datasets, data services, general services, events, concepts, and information models. It consumes metadata in RDF format, enriches the content, and transforms it into formats that are better suited for frontend applications and search indexing.
Portal is a system for searching and browsing metadata related to datasets, data services, general services, events, concepts, and information models. It consumes metadata from Harvesting and builds various indexes and databases to support:
- Full-text search
- Semantic search (e.g., using RAG)
- SPARQL queries
- And more advanced discovery features
Portal also integrates metadata quality scores from the Metadata Quality system to help users assess the quality of the information they are viewing.
Metadata Quality is a system for computing quality scores on metadata about datasets. It consumes harvested datasets and evaluates them based on predefined criteria to produce a quality score.
This score is then used by the Portal to present an indication of metadata quality to end users.
IAM is a system for handling authentication and authorization across the platform. It integrates with multiple identity providers and supports the OpenID Connect protocol to enable secure and flexible user access.
IAM ensures that users are properly authenticated and that access to features or data is controlled based on roles and permissions.
