From 45d53caa5c4d439676f38a520e8a517d9fb5f758 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Wed, 22 May 2024 16:35:58 +1000 Subject: [PATCH 1/8] Link to quesnelia docs.f.o --- source-code/map.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source-code/map.md b/source-code/map.md index 9d344a2f..3e000b7d 100644 --- a/source-code/map.md +++ b/source-code/map.md @@ -13,7 +13,7 @@ menuSubIndex: 2 List of FOLIO modules hosted at GitHub, with links to some related documentation. Also refer to the [Source-code overview](/source-code/). -{% assign versionUserDocs = "poppy" %} +{% assign versionUserDocs = "quesnelia" %} {% assign urlGithub = "https://github.com" %} {% assign urlApiBase = "https://dev.folio.org/reference/api" %} {% assign urlApiBaseLocal = "/reference/api" %} From 2d983737d905107cdbe3b3b7d1057b6d5e79cb7a Mon Sep 17 00:00:00 2001 From: David Crossley Date: Wed, 22 May 2024 16:37:29 +1000 Subject: [PATCH 2/8] Include more remote content FOLIO-3311 --- _data/remote-docs.json | 1 + 1 file changed, 1 insertion(+) diff --git a/_data/remote-docs.json b/_data/remote-docs.json index c938352e..07fd6541 100644 --- a/_data/remote-docs.json +++ b/_data/remote-docs.json @@ -20,6 +20,7 @@ "https://raw.githubusercontent.com/folio-org/rfcs/master/text/0002-kafka-tenant-collection-topics.md", "https://raw.githubusercontent.com/folio-org/rfcs/master/text/0003-folio-breaking-changes.md", "https://raw.githubusercontent.com/folio-org/rfcs/master/text/0004-iso-8601.md", + "https://raw.githubusercontent.com/folio-org/rfcs/master/text/0006-folio-distributed-vs-centralized-configration.md", "https://raw.githubusercontent.com/folio-org/stripes-cli/master/doc/dev-guide.md", "https://raw.githubusercontent.com/folio-org/stripes-cli/master/doc/user-guide.md", "https://raw.githubusercontent.com/folio-org/stripes-components/master/README.md", From 9be2421f07af39e18989809fad7ce67942402537 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Wed, 22 May 2024 16:43:31 +1000 Subject: [PATCH 3/8] Update remote content FOLIO-3311 --- _remote/edge-common/README.md | 56 ++++++++--- _remote/raml-module-builder/README.md | 4 +- ...distributed-vs-centralized-configration.md | 97 +++++++++++++++++++ 3 files changed, 142 insertions(+), 15 deletions(-) create mode 100644 _remote/rfcs/text/0006-folio-distributed-vs-centralized-configration.md diff --git a/_remote/edge-common/README.md b/_remote/edge-common/README.md index 49ecb58f..f551d922 100644 --- a/_remote/edge-common/README.md +++ b/_remote/edge-common/README.md @@ -152,22 +152,52 @@ e.g. Key=`ab73kbw90e/diku`, Field=`diku` Configuration information is specified in two forms: 1. System Properties - General configuration -1. Properties File - Configuration specific to the desired secure store +2. Properties File - Configuration specific to the desired secure store ### System Properties -Property | Default | Description ------------------------- | ----------- | ------------- -`port` | `8081` | Server port to listen on -`okapi_url` | *required* | Where to find Okapi (URL) -`secure_store` | `Ephemeral` | Type of secure store to use. Valid: `Ephemeral`, `AwsSsm`, `Vault` -`secure_store_props` | `NA` | Path to a properties file specifying secure store configuration -`token_cache_ttl_ms` | `3600000` | How long to cache JWTs, in milliseconds (ms) -`null_token_cache_ttl_ms`| `30000` | How long to cache login failure (null JWTs), in milliseconds (ms) -`token_cache_capacity` | `100` | Max token cache size -`log_level` | `INFO` | Log4j Log Level -`request_timeout_ms` | `30000` | Request Timeout -`api_key_sources` | `PARAM,HEADER,PATH` | Defines the sources (order of precendence) of the API key. +| Property | Default | Description | +|---------------------------|---------------------|---------------------------------------------------------------------------| +| `port` | `8081` | Server port to listen on | +| `okapi_url` | *required* | Where to find Okapi (URL) | +| `secure_store` | `Ephemeral` | Type of secure store to use. Valid: `Ephemeral`, `AwsSsm`, `Vault` | +| `secure_store_props` | `NA` | Path to a properties file specifying secure store configuration | +| `token_cache_ttl_ms` | `3600000` | How long to cache JWTs, in milliseconds (ms) | +| `null_token_cache_ttl_ms` | `30000` | How long to cache login failure (null JWTs), in milliseconds (ms) | +| `token_cache_capacity` | `100` | Max token cache size | +| `log_level` | `INFO` | Log4j Log Level | +| `request_timeout_ms` | `30000` | Request Timeout | +| `api_key_sources` | `PARAM,HEADER,PATH` | Defines the sources (order of precendence) of the API key. | + +### System Properties for TLS configuration for Http server +To configure Transport Layer Security (TLS) for the HTTP server in an edge module, the following configuration parameters should be used. +Parameters marked as Required are required only in case when ssl_enabled is set to true. + +| Property | Default | Description | +|-----------------------------------|-------------------|---------------------------------------------------------------------------------------------| +| `http-server.ssl_enabled` | `false` | Set whether SSL/TLS is enabled for Vertx Http Server | +| `http-server.keystore_type` | `NA` | (Required). Set the type of the keystore. Common types include `JKS`, `PKCS12`, and `BCFKS` | +| `http-server.keystore_provider` | `NA` | Set the provider name of the key store | +| `http-server.keystore_path` | `NA` | (Required). Set the location of the keystore file in the local file system | +| `http-server.keystore_password` | `NA` | (Required). Set the password for the keystore | +| `http-server.key_alias` | `NA` | Set the alias of the key within the keystore. | +| `http-server.key_alias_password` | `NA` | Optional param that points to a password of `key_alias` if it protected | + +### System Properties for TLS configuration for Web Client +To configure Transport Layer Security (TLS) for Web clients in the edge module, you can use the following configuration parameters. +Truststore parameters for configuring Web clients are optional even when ssl_enabled = true. +If truststore parameters need to be populated, truststore_type, truststore_path and truststore_password are required. + +| Property | Default | Description | +|-----------------------------------|-------------------|----------------------------------------------------------------------------------| +| `web-client.ssl_enabled` | `false` | Set whether SSL/TLS is enabled for Vertx Http Server | +| `web-client.truststore_type` | `NA` | Set the type of the keystore. Common types include `JKS`, `PKCS12`, and `BCFKS` | +| `web-client.truststore_provider` | `NA` | Set the provider name of the key store | +| `web-client.truststore_path` | `NA` | Set the location of the keystore file in the local file system | +| `web-client.truststore_password` | `NA` | Set the password for the keystore | +| `web-client.key_alias` | `NA` | Set the alias of the key within the keystore. | +| `web-client.key_alias_password` | `NA` | Optional param that points to a password of `key_alias` if it protected | + ## Additional information diff --git a/_remote/raml-module-builder/README.md b/_remote/raml-module-builder/README.md index be868f53..be6a1ca2 100644 --- a/_remote/raml-module-builder/README.md +++ b/_remote/raml-module-builder/README.md @@ -1611,7 +1611,7 @@ For each **table** in `tables` property: * `arraySubfield` - is the key of the object that is used for the primary term when array relation modifiers are in use. This is typically also defined when `arrayModifiers` are also defined. * `multiFieldNames` - see [CQL2PgJSON: Multi Field Index](#cql2pgjson-multi-field-index) above * `sqlExpression` - see [CQL2PgJSON: Multi Field Index](#cql2pgjson-multi-field-index) above - * `sqlExpressionQuery` - How to wrap the query string $ before matching against the index expression. This overwrites any `caseSensitive` and `removeAccents` wrapping. Example that manually implements case insensitive matching: `"sqlExpression": "lower(jsonb->>'name')", "sqlExpressionQuery": "lower($)"` + * `sqlExpressionQuery` - How to wrap the query string $ before matching against the index expression. This overwrites any `caseSensitive` and `removeAccents` wrapping. Example that manually implements case insensitive matching: `"sqlExpression": "lower(jsonb->>'name')", "sqlExpressionQuery": "lower($)"`. Example for plain field, note the required braces: `"sqlExpression": "(jsonb->>'name')", "sqlExpressionQuery": "$"`. * Do not manually add an index for an `id` field or a foreign key field. They get indexed automatically. 7. `ginIndex` - generate an inverted index on the JSON using the `gin_trgm_ops` extension. Allows for left and right truncation LIKE queries and regex queries to run in an optimal manner (similar to a simple search engine). Note that the generated index is large and does not support the full field match (SQL `=` operator and CQL `==` operator without wildcards). See the `likeIndex` for available options. 8. `uniqueIndex` - create a unique index on a field in the JSON @@ -1713,7 +1713,7 @@ The fields in the **script** section include: 1. `run` - either `before` or `after` the tables / views are generated 2. `snippet` - the SQL to run. There is no variable replacement. Use `snippetPath` if variable replacement or line breaks for readability are needed. 3. `snippetPath` - relative path to a file with SQL script to run. If `snippetPath` is set then `snippet` field will be ignored. -4. `fromModuleVersion` - same as `fromModuleVersion` for table. If no fromModuleVersion is provided the SQL runs on each upgrade. This is reasonable if the SQL runs fast and is idempotent, for example ```CREATE OR REPLACE FUNCTION``` and you want to avoid the error-prone fromModuleVersion value maintenance needed when changing the function. +4. `fromModuleVersion` - same as `fromModuleVersion` for table. If no fromModuleVersion is provided the SQL runs on each upgrade. This is reasonable if the SQL runs fast and is idempotent, for example ```CREATE OR REPLACE FUNCTION``` and you want to avoid the error-prone fromModuleVersion value maintenance needed when changing the function. The SQL also runs on the initial install of a tenant. A `snippetPath` SQL file (but not a `snippet` SQL statement) can make use of [FreeMarker template engine](https://freemarker.apache.org) that runs and evaluates on runtime. For examples, see RMB's diff --git a/_remote/rfcs/text/0006-folio-distributed-vs-centralized-configration.md b/_remote/rfcs/text/0006-folio-distributed-vs-centralized-configration.md new file mode 100644 index 00000000..a6cd6c1b --- /dev/null +++ b/_remote/rfcs/text/0006-folio-distributed-vs-centralized-configration.md @@ -0,0 +1,97 @@ +--- +layout: null +--- + +- Contributors: + - [Julian Ladisch](julian.ladisch@gbv.de) + - [Florian Gleixner](Florian.Gleixner@lrz.de) +- RFC PRs: + - PRELIMINARY REVIEW: https://github.com/folio-org/rfcs/pull/23 + - DRAFT REVIEW: https://github.com/folio-org/rfcs/pull/28 + - PUBLIC REVIEW: https://github.com/folio-org/rfcs/pull/28 + - FINAL REVIEW: TBD +- Outcome: (Leave this blank. Will eventually be either ACCEPTED or REJECTED) + + +# Folio distributed vs. centralized configuration + +## Summary + +Drop mod-configuration module. Favour distributed configuration over centralized configuration, provide guidelines for storage of configuration values. + +## Motivation + +FOLIO relies on a tenant aware extendable microservice system. It is highly configurable and adoptable on all kind of requirements. FOLIO had no strict guidelines for developers, how and where to store the configuration values. This led to a situation where different approaches where developed by different developer teams. +This makes it difficult for implementers, administrators and users to configure the system appropriate, as they may have to search the documentation how to read or write these configuration values. +Basically there are two competing concepts where to store the values: distributed in the owning modules or centralized in a special module that offers a configuration store for other modules. There exist two solutions for centralized configurations: mod-settings and mod-configuration (deprecated, to be dropped). +This RFC shall give guidelines for developers, where configuration values shall be set. + +## Detailed Explanation/Design + +This RFC deals with configurations that configure the behaviour of the Folio tenant. Among all different kinds of configurations, this RFC does not deal with the following configurations, as their location and method to set/get them shall not be changed: + +* Settings stored in Okapi's /_/env APIs. +* Settings stored in Infrastructure (Kubernetes / Rancher) config maps and secrets. +* Settings stored in module container environment variables. +* Settings stored in the stripes front-end (stripes.config.js, etc.) + +### Use of mod-configuration will be discontinued after Ramsons release + +mod-configuration is deprecated due to security problems since March 2022. It shall not be used any more to add new configuration variables. Modules still using mod-configuration have to move to other solutions after the Ramsons release. + +### Distributed configuration is preferred + +Distributed configuration means, that each module stores its configuration values itself, and offers API endpoints to query and store these values. Distributed configuration in a microservice architecture has some advantages: + +* The modules can validate the values according to format and dependencies +* Modules do not depend on a configuration module, hence a better separation of microservices can be achieved +* Since all API endpoints have to be documented, a basic documentation of possible configuration variables is mandatory +* Configuration values can be cached, since no other module can change values. +* Access to configuration values can effectively controlled by permissions defined in the module. +* Write-only configuration values are possible, like for credentials. The module can offer other operators than reading values like comparing hashes (possible in central configuration too?) +* Modules can handle upgrade of configuration variable names or values during module upgrades more flexible + +Even when there are also some drawbacks on distributed configuration, it is the preferred way to configure backend modules in FOLIO. + +### When to use central configuration + +mod-settings solves the security problems of mod-configuration. It is the preferred module if configuration variables shall be stored centrally. It is not recommended to develop specialized modules for other central configuration store. + +Centralized configuration can only be used for either: + +* Non-sensitive information, that are used by many modules or are completely independent of any module. One example are locale settings. +* Configurations that are specific to a user. + +While these configurations can also be stored decentralized in a module, the developer can decide where these values shall be stored. + +### Migration + +See Timing Section below. + +## Risks and Drawbacks + +Migrations from old mod-configuration can fail, and therefore tenant upgrades may fail. +Distributed configuration requires more developer effort than central configuration. + +## Rationale and Alternatives + +Both central and distributed configurations were discussed. +While pure central configuration offers easier access to the complete configuration of a tenant, this is not desireable since: +* it is an antipattern in a microservice architecture ([shared persistence/data ownership](https://arxiv.org/ftp/arxiv/papers/1908/1908.04101.pdf)) +* validation of values is not possible +* differentiating configuration data and application data is not easy: are circulation rules configuration data or application data? + +A pure distributed configuration has the following drawbacks: +* for centrally used configuration values, like locale settings, an owning module has to be found. It may not be intuitive finding out, which module holds these configurations. + +Therefore a distributed configuration with some exceptions has been considered. + +## Timing + +Since most modules already store configuration values in a distributed way, only some cases need to be addressed. +For locale properties and other properties still residing exclusively in mod-configuration, the access to these properties has to be moved to the module (distributed configuration, preferred) or to the mod-settings (centralized configuration, not preferred) after the Ramsons release. Therefore a mod-configuration module offering only READ and DELETE APIs will run in Sunflower and the modules still using mod-configuration have to transfer their properties to mod-settings or to a distributed configuration. Migrated configurations in mod-configuration have to be deleted. This has to be done during module upgrades. +mod-configuration will be removed in the release following the Sunflower release. + +## Unresolved Questions + +None at the moment. From 39890fa42fbd49b91de8c5ce357ca6b0632163fc Mon Sep 17 00:00:00 2001 From: David Crossley Date: Wed, 22 May 2024 16:52:54 +1000 Subject: [PATCH 4/8] Link JDK17 TC wiki notes --- _faqs/how-to-specify-backend-java-ci.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/_faqs/how-to-specify-backend-java-ci.md b/_faqs/how-to-specify-backend-java-ci.md index 7a9c8ee3..b0e59e7f 100644 --- a/_faqs/how-to-specify-backend-java-ci.md +++ b/_faqs/how-to-specify-backend-java-ci.md @@ -9,7 +9,7 @@ faqOrder: 12 Back-end Java-based modules can specify which build image to use during the Jenkins continuous integration builds. -For front-end modules that still use Jenkins build rather than GitHub Workflows, the buildNode provides the relevant Nodejs version. +For the few front-end modules that still use Jenkins build rather than GitHub Workflows, the buildNode provides the relevant Nodejs version. ## Using Java 17 @@ -20,10 +20,11 @@ For projects that use Java 17: * In [Jenkinsfile](/guides/jenkinsfile/), declare the mandatory module heath check endpoint via the Docker command `healthChkCmd` to use 'wget'. (There is no 'curl' available on the base image.) * In Dockerfile, use:\ `FROM folioci/alpine-jre-openjdk17:latest` +* See other notes (e.g. POM file, and dependencies, and wget example) at Wiki [JDK 17 and Java 17](https://folio-org.atlassian.net/wiki/x/Th9N). ## Using Java 11 -For projects that use Java 11: +For some projects that use Java 11 (there should now not be any): * In [Jenkinsfile](/guides/jenkinsfile/), declare this in the "buildMvn" section:\ `buildNode = 'jenkins-agent-java11'` From 0bb1322cfd12707e231609f4a9f0aa10236b4bc5 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Thu, 23 May 2024 11:53:17 +1000 Subject: [PATCH 5/8] Confirm Slack rancher-support --- _faqs/how-to-get-started-with-rancher.md | 1 + guidelines/which-forum.md | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/_faqs/how-to-get-started-with-rancher.md b/_faqs/how-to-get-started-with-rancher.md index f6e9a35d..47006a2f 100644 --- a/_faqs/how-to-get-started-with-rancher.md +++ b/_faqs/how-to-get-started-with-rancher.md @@ -27,6 +27,7 @@ Some teams also provide [further general instructions](/guides/run-local-folio/# For general assistance use the [Slack channel](/guidelines/which-forum/#slack) `#rancher-support` and help via your team-specific channel. To raise Jira tickets related to Rancher scratch environments, refer to the [FAQ](/faqs/how-to-raise-devops-ticket/#rancher-scratch-environments). +This is needed when contacting the `#rancher-support` Slack channel. See the Wiki [How to work with Rancher environments](https://folio-org.atlassian.net/wiki/spaces/FOLIJET/pages/1396559/How+to+work+with+Rancher+environments). diff --git a/guidelines/which-forum.md b/guidelines/which-forum.md index f7181111..ae587230 100644 --- a/guidelines/which-forum.md +++ b/guidelines/which-forum.md @@ -157,7 +157,7 @@ and in the [Guidelines for FOLIO issue tracker](/guidelines/issue-tracker/#linki - A place for heads-up type of notices. -- There are now many channels. Browse the list, via the Slack application, to review and join any appropriate ones. +- There are now many channels. Browse the list via the Slack application, to review and join any appropriate ones. Some are not public, so you will need to be invited. Some relevant channels for developers are: - `#support` -- Any topic needing assistance. @@ -169,7 +169,7 @@ and in the [Guidelines for FOLIO issue tracker](/guidelines/issue-tracker/#linki - `#raml-module-builder` -- Specific discussion for RMB development. - `#stripes` -- Specific discussion for front-end development. - `#devops` -- Assistance with FOLIO infrastructure, requests to add new GitHub Team members, etc. - - `#scratch-environments` -- Assistance with the team-specific [Rancher developer scratch environments](/faqs/how-to-get-started-with-rancher/). + - `rancher-support` (was previously known as `#scratch-environments`) -- Assistance with the team-specific [Rancher developer scratch environments](/faqs/how-to-get-started-with-rancher/). - `#folio-ci` -- Notifications from the Jenkins CI. See usage [notes](/guides/navigate-commits/). Mute this channel and visit when needed. - Use `@mentions` with care. This is especially important on broad channels such as `#general`, as using `@channel` there will send notifications to many hundreds of people. From 39a9592c5977d26bf2148b7d81d7ee5670d878c4 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Thu, 23 May 2024 12:00:26 +1000 Subject: [PATCH 6/8] Update list of repositories FOLIO-359 --- _data/repos.json | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/_data/repos.json b/_data/repos.json index 20151cc2..434cd6e1 100644 --- a/_data/repos.json +++ b/_data/repos.json @@ -1,6 +1,6 @@ { "metadata": { - "generatedDateTime": "2024-04-23T23:55:39.713613+00:00" + "generatedDateTime": "2024-05-22T06:26:22.691494+00:00" }, "repos": [ { @@ -299,7 +299,7 @@ "ramlDirName": null, "repoLanguageHint": "java", "repoType": "backend-edge", - "snippetIntro": "

Provides an ability to retrieve license term information

", + "snippetIntro": "

Provides an ability to retrieve license term information from FOLIO to external systems. The data flow is edge-erm <-> FOLIO <-> mod-agreements.

", "workflows": [ "api-doc.yml", "api-lint.yml", @@ -2653,7 +2653,7 @@ "apiTypes": "OAS", "defaultBranch": "master", "description": "Module manages record specifications and validates records against them", - "docDirName": null, + "docDirName": "docs", "hintOas": true, "isSpringway": true, "name": "mod-record-specifications", @@ -2661,11 +2661,12 @@ "ramlDirName": null, "repoLanguageHint": "java", "repoType": "backend-mod", - "snippetIntro": "

Module for managing and validating records against predefined specifications.

", + "snippetIntro": "

mod-record-specifications is a module for managing and validating records against predefined specifications.

", "workflows": [ "api-doc.yml", "api-lint.yml", "api-schema-lint.yml", + "build-dependants.yml", "spectral-lint.yml" ], "workflowsFolio": [ @@ -4503,6 +4504,9 @@ "snippetIntro": "

The Lists UI Module, or ui-lists, is a Stripes UI Module used to create and manage Lists which are queries across multiple modules (for example: find all items marked as missing or declared lost, or find all open loans assigned to inactive users).

", "workflows": [ "ui.yml" + ], + "workflowsFolio": [ + "ui.yml" ] }, { @@ -4677,12 +4681,10 @@ "repoType": "ui-plugin", "snippetIntro": "

This package furnishes a single Stripes plugin of type bursar-export, which can be included in Stripes modules by means of a <Pluggable type='bursar-export'> element. See the Plugins section of the Module Developer's Guide.

", "workflows": [ - "build-npm-release.yml", - "build-npm.yml" + "ui.yml" ], "workflowsFolio": [ - "build-npm-release.yml", - "build-npm.yml" + "ui.yml" ] }, { From 4b8bb3498a947b5995c8895cd8d9e73e9c79f851 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Thu, 23 May 2024 12:18:24 +1000 Subject: [PATCH 7/8] Emphasise section Platform hourly build and the need to verify --- guides/automation.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/guides/automation.md b/guides/automation.md index 9b45cbd0..41f08ec6 100644 --- a/guides/automation.md +++ b/guides/automation.md @@ -48,7 +48,6 @@ Similarly okapi can be accessed via `folio-snapshot-core-okapi` (and see notes i --> The folio-snapshot and folio-snapshot-2 are constructed identically every 24 hours, with a 12-hour offset. - There are two [flower release](/guides/regular-releases/) environments (the current and the previous) which are rebuilt weekly on a Sunday. The environments are completely torn-down and rebuilt again. @@ -81,14 +80,6 @@ Correlate the "build number" with that shown in the output log of the project's The server is built every day, to finish about 02:03 UTC.\ (See Jenkins job: [folio-snapshot](https://jenkins-aws.indexdata.com/job/FOLIO_Reference_Builds/job/folio-snapshot/) which starts about 01:24 UTC.). -#### Platform hourly build - -The set of frontend modules are those listed in the "snapshot" branch of the Stripes Platform. - -The "snapshot" branch of the Stripes Platform is rebuilt every hour, starting about 19 minutes past the hour and finishing about 50 minutes past (see Jenkins job: [build-platform-complete-snapshot](https://jenkins-aws.indexdata.com/job/Automation/job/build-platform-complete-snapshot/)). -If successful, then this will [regenerate](/guidelines/release-procedures/#add-to-platforms) the yarn.lock and install files of the Platform (see [note above](#install-json)), to be utilised by the abovementioned "folio-snapshot" job. -So if there is an urgent need to [rebuild](#off-schedule-rebuilds) "folio-snapshot" outside of normal automation, so as to include a new snapshot of a module, then this build needs to have run before the "folio-snapshot" build is re-run. - #### Included module versions The folio-snapshot builds consist of the master branch of each frontend module at that time, paired with the latest version of backend modules that meet the dependency requirements of the frontend (as determined by the preceding [hourly platform build](#platform-hourly-build)). @@ -105,6 +96,16 @@ The server is built every day, to finish about 14:00 UTC.\ The folio-snapshot-latest is an alias for folio-snapshot-2. +### Platform hourly build + +The set of frontend modules are those listed in the "snapshot" branch of the Stripes Platform. + +The "snapshot" branch of the Stripes Platform is rebuilt every hour, starting about 19 minutes past the hour and finishing about 50 minutes past (see Jenkins job: [build-platform-complete-snapshot](https://jenkins-aws.indexdata.com/job/Automation/job/build-platform-complete-snapshot/)). +If successful, then this will [regenerate](/guidelines/release-procedures/#add-to-platforms) the yarn.lock and install files of the Platform (see [note above](#install-json)), to be utilised by the abovementioned "folio-snapshot" job. +So if there is an urgent need to [rebuild](#off-schedule-rebuilds) "folio-snapshot" outside of normal automation, so as to include a new snapshot of a module, then this build needs to have run before the "folio-snapshot" build is re-run. + +It is vitally important that developers ensure success of the subsequent hourly build following any major changes that they merge to mainline (especially on a Friday afternoon). As noted in the previous paragraph, failures of this hourly job will cause the "folio-snapshot" builds to use out-of-date install files. + ### folio-orchid [https://folio-orchid.dev.folio.org/](https://folio-orchid.dev.folio.org/) From becc5cbadab2ffc774ef34d6be1d87a09ff33613 Mon Sep 17 00:00:00 2001 From: David Crossley Date: Thu, 23 May 2024 12:52:37 +1000 Subject: [PATCH 8/8] Add reference environments Troubleshooting --- _faqs/how-to-obtain-refenv-logs.md | 2 ++ guides/automation.md | 15 ++++++++++++++- 2 files changed, 16 insertions(+), 1 deletion(-) diff --git a/_faqs/how-to-obtain-refenv-logs.md b/_faqs/how-to-obtain-refenv-logs.md index c71503a7..13a5b725 100644 --- a/_faqs/how-to-obtain-refenv-logs.md +++ b/_faqs/how-to-obtain-refenv-logs.md @@ -22,3 +22,5 @@ For example notify yourself, or another person, or notify your team channel. All members of folio-org are configured to run this job. The service creates a tar file containing the specified logs and uploads to an S3 bucket. A URL to retrieve the logs is returned via a Slack notification. +For failed [Platform hourly build](/guides/automation/#platform-hourly-build) jobs, the Okapi log is linked from the left-hand panel of its Jenkins job summary page. + diff --git a/guides/automation.md b/guides/automation.md index 41f08ec6..8a542712 100644 --- a/guides/automation.md +++ b/guides/automation.md @@ -48,7 +48,7 @@ Similarly okapi can be accessed via `folio-snapshot-core-okapi` (and see notes i --> The folio-snapshot and folio-snapshot-2 are constructed identically every 24 hours, with a 12-hour offset. -There are two [flower release](/guides/regular-releases/) environments (the current and the previous) which are rebuilt weekly on a Sunday. +There are also two [flower release](/guides/regular-releases/) environments (the current and the previous) which are rebuilt weekly on a Sunday. The environments are completely torn-down and rebuilt again. @@ -73,6 +73,18 @@ then co-ordinate that on the Slack channel #hosted-reference-envs in that build's automatically generated [install.json](https://github.com/folio-org/platform-complete/blob/snapshot/install.json) file. Correlate the "build number" with that shown in the output log of the project's "Publish module descriptor" stage. +### Troubleshooting + +Failures with the main builds are automatically sent to the `#hosted-reference-envs` Slack channel. + +Follow its links to the Jenkins job output. +Refer to the FAQs +[How to obtain reference environment module logs](/faqs/how-to-obtain-refenv-logs/) +and +[How to investigate Jenkins build logs](/faqs/how-to-investigate-jenkins-logs/) + +As noted in the [Platform hourly build](#platform-hourly-build) section below, it is vitally important that developers ensure success of the subsequent hourly build following any major changes that they merge to mainline (especially on a Friday afternoon). Continued failures of this hourly job will cause the “folio-snapshot” builds to use out-of-date install files. + ### folio-snapshot [https://folio-snapshot.dev.folio.org/](https://folio-snapshot.dev.folio.org/) @@ -105,6 +117,7 @@ If successful, then this will [regenerate](/guidelines/release-procedures/#add-t So if there is an urgent need to [rebuild](#off-schedule-rebuilds) "folio-snapshot" outside of normal automation, so as to include a new snapshot of a module, then this build needs to have run before the "folio-snapshot" build is re-run. It is vitally important that developers ensure success of the subsequent hourly build following any major changes that they merge to mainline (especially on a Friday afternoon). As noted in the previous paragraph, failures of this hourly job will cause the "folio-snapshot" builds to use out-of-date install files. +Refer to the [Troubleshooting](#troubleshooting) assistance section above. ### folio-orchid