2022.2.11_7

This release implements a few improvements:

• Dockerfile has been updated so that HTML documentation is built together with the service, and made accessible via /static/docs/index.html. (In case of current dev instance, the URL is http://dev.bibxml.org/static/docs/index.html.)
• The bibliographic item details page shows export links for all registered export formats, rather than just bibxml.
• When clicking on a particular document ID in search results listing, the anchor will be used to automatically scroll to the bibliographic item with that identifier, if there are multiple items shown on the page.
• Management section that lists available indexable datasets shows progress information without having to navigate to individual dataset’s page.

2022.2.11_6

Datatracker OAuth

This release introduces working Datatracker OAuth2 integration:

• Log in with Datatracker is now shown, if client ID and secret are configured.
• API now accepts not just the Datatracker development API token, but also recognizes OAuth credentials in user’s session. If access token is valid and not expired (this requires a trip to Datatracker), the function is called.
• XML & JSON download links are now displayed again for users logged in via Datatracker, making use of API recognizing OAuth session. (#113)

Notable limitations

• The name of the logged-in user is not displayed correctly (#116)
• Access token refresh is not implemented (#117)
• If Datatracker OAuth is expired at the time of JSON or XML download, the downloaded file will contain an error message (#118)

xml2rfc path handling

xml2rfc compatibility was extended to support manual mappings. Mappings dictate which docid.id a given xml2rfc-style path must correspond to. Mappings are stored in DB.

Note: existence of a mapping does not mean the path is always resolved; if a document with mapped docid.id does not exist in the index, the mapping will fail and “auto” or “fallback” method will be used.

Management GUI now includes an xml2rfc compatibility dashboard (https://dev.bibxml.org/management/xml2rfc-compat/). When a directory is selected, it shows a list of XML files inside. If there are more than 6000 files, user agent must have JavaScript turned on, and data windowing is used in that case (GUI becomes unusable with many thousands of DOM entries).

• Clicking a file allows to create/edit/remove its manual mapping, and show if path cannot resolve.
• For each file visible within the scrollable view, the effective path resolution method is dynamically loaded and displayed (this requires JavaScript). It can be one of the following, in order of preference:

  • method	behavior
    manual	if manual map was specified, try to find an item with mapped docid.id
    auto	if manual map does not exist or failed to resolve, use an automatic fetcher function that tries to construct a JSON query against Relaton bibliographic data based on XML filename in the path
    fallback	if neither of the above worked, proxy to data indexed from xml2rfc tools mirror Git repository
    N/A	if the fallback failed as well (e.g. the xml2rfc mirror source was not indexed or an error occurred during request)

• For a selected file it’s possible (with JavaScript enabled) to test resolution and view the resulting XML and resolution methods.
  • It allows to compare the XML against currently existing xml2rfc tools XML, but that JavaScript request currently fails due to cross-origin restrictions.

Other

GUI was updated in places. Bibliographic item detail view is now a bit more compact with some copy changes, and more properties (authors, publishers) now link to structured searches in addition to relations. Datatracker login block is more prominent.

Some bugs were fixed, including API for resolving DOI.

2022.2.3_2

This release primarily deals with sourcing.

• Bibliographic data sources are now pluggable. Main Relaton sources are registered in main.sources, and xml2rfc makes use of it as well for indexing xml2rfc mirror repository for fallback purposes.
  • The associated refactor moved source-related functionality out from management module into either sources or main. Python module reference in documentation has been updated to reflect the current architecture.
• xml2rfc-style paths now support fallback when up-to-date equivalent was not found for given anchor. This requires new xml2rfc source to be indexed (which can be done manually via management GUI, or via index API endpoint).

Other changes include:

• Container’s Dockerfile was updated to make better use of layer caching
• Monitoring services were split into auxiliary Compose configuration that can be used in conjunction with the main one via multiple -f options
• A possible crash when searching bibliographic data via API has been fixed
• API now conforms to the advertised spec better by returning JSON responses in case of bad request or server errors

2022.2.1

New features in this release mostly concern themselves with metrics, tracking and integrations:

• Matomo tracker or tag manager are now supported via MATOMO_URL, and MATOMO_CONTAINER and/or MATOMO_SITE_ID environment variables. (Needs more integration testing.)
• Metrics for bibliographic data accesses via GUI and API are now exported in Prometheus format under /metrics path.
• The service now allows site visitors to log in via Datatracker, showing the requisite widget if DATATRACKER_CLIENT_ID and DATATRACKER_CLIENT_SECRET environment variables are provided at deploy time. (#69)
  • OAuth2 flow is implemented, but needs further integration testing and clarification of the user info endpoint schema.
  • Callback URL is expected to be https://<hostname>/datatracker-auth/callback/
• Multiple additional API secrets can now be provisioned for management API and management GUI access via EXTRA_API_SECRETS environment variable. (#68)

Added containers

There are new services in the bundled Compose file configuration.

• A container that exports Celery-level metrics in Prometheus format by monitoring Redis backend.
• A pre-configured Prometheus instance monitoring three targets (web server, and two targets for async task runner).
• A Grafana instance provisioned with Prometheus as data source and a couple of dashboards for monitoring API and GUI bibliographic data access metrics.

Notable regressions

• We now explicitly run only one worker per container, both in case of Hypercorn serving web GUI and in case of Celery running async task worker. CPU_COUNT variable is no longer supported.

  If needed, web and celery containers themselves can be scaled horizontally. In such custom configuration it’s important to also ensure Prometheus is set up to discover scaled instances in order to import and aggregate bibliographic data access statistics from all of them. Bundled Compose configuration does not support that scenario at this time.

• Links to get bibliographic item data in Relaton JSON or XML are not shown to site visitors for the time being, as they were simply linking to API endpoints and API now requires header-based authentication—so those links wouldn’t work anymore.

  They can be reenabled in future if necessary, using a different mechanism that e.g. relies on Datatracker OIDC authentication.

Other changes

Various smaller fixes and improvements include:

• List of indexable datasets in OpenAPI spec for management endpoints is now generated dynamically and no longer erroneously includes doi
• Search query formats, changed some time back, are now documented in OpenAPI spec
• Search widget on landing page now features help explaining different query methods
• json_struct method has been added to web GUI search widget
• Version displayed in OpenAPI specs and built documentation now reflects current codebase version (such as 2022.2.1 for this release)
• and more.

2022.1.29_2

This release affects OpenAPI specifications:

• OpenAPI spec for xml2rfc-style paths was brought up to date. It no longer lists all the directories, but bibxml to bibxml9 are covered.
• Bibliographic item schema definitions in OpenAPI specification are now automatically generated from Pydantic model definitions (they can be explored in application/json response schema under retrieval endpoint).
• OpenAPI specs no longer refer to bibliographic items as “citations”.

2022.1.29_1

This release makes Datatracker’s bibxml API key required and validated on each request to bibliographic item data search and retrieval endpoints (#69), and lays foundation for further Datatracker integration (such as retrieving latest Internet Draft revisions).

It also removes accidental trailing dots or .xml extension when using web GUI’s simple search widget as a minor convenience measure.