Merge pull request #430 from FirelyTeam/release/fs-5.6.0

Release/fs 5.6.0
FirelyTeam · Apr 11, 2024 · cf73beb · cf73beb
2 parents 56c9b3a + 6378158
commit cf73beb
Show file tree

Hide file tree

Showing 12 changed files with 155 additions and 37 deletions.
diff --git a/compliance/igs.rst b/compliance/igs.rst
@@ -147,7 +147,7 @@ In summary, the USCDI defines the core health data elements for nationwide inter
      - * `hl7.fhir.us.core|3.1.1 <https://registry.fhir.org/package/hl7.fhir.us.core|3.1.1>`_
        * `hl7.fhir.us.core|4.0.0 <https://registry.fhir.org/package/hl7.fhir.us.core|4.0.0>`_
        * `hl7.fhir.us.core|5.0.1 <https://registry.fhir.org/package/hl7.fhir.us.core|5.0.1>`_
-       * `hl7.fhir.us.core|5.0.1 <https://registry.fhir.org/package/hl7.fhir.us.core|6.1.0>`_
+       * `hl7.fhir.us.core|6.1.0 <https://registry.fhir.org/package/hl7.fhir.us.core|6.1.0>`_
 
 Known Limitations
 -----------------

diff --git a/features_and_tools/firely-server-ingest.rst b/features_and_tools/firely-server-ingest.rst
@@ -183,6 +183,8 @@ Supported arguments
 |                                                          |                                     |          | When using the command line argument, the entries of the array must be provided one by one by suffixing with the relevant index. For example:       |
 |                                                          |                                     |          | ``--urlConvElems:0 some.path  --urlConvElems:1 some.other.path``                                                                                    |
 +----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
+| ``--useRecoveryJournal <recoveryJournalDirectory>``      | recoveryJournalDirectory            |          | A directory containing the recovery journal                                                                                                         |
++----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``--mongoCollection <mongoCollection>``                  | mongodb/entryCollection             |          | Collection name for entries                                                                                                                         |
 +----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``--mongoConnectionstring <connectionstring>``           | mongodb/connectionString            | yes      | Connection string to Firely Server MongoDb database                                                                                                 |
@@ -204,8 +206,6 @@ Supported arguments
 +----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``--sqlExistQryPar <sqlExistQryPar>``                    | sqlserver/queryExistenceParallel    |          | The number of parallel threads querying the DB to check whether a resource exists (only when ``--update-existing-resources`` is set to false).      |
 +----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
-| ``--readPar <readPar>``                                  | workflow/readParallel               |          | Number of threads to read from the source. Reading is quite fast so it need not be high                                                             |
-+----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``--readBuffer <readBuffer>``                            | workflow/readBufferSize             |          | Number of resources to buffer after reading                                                                                                         |
 +----------------------------------------------------------+-------------------------------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------+
 | ``--metaPar <metaPar>``                                  | workflow/metaParallel               |          | Number of threads to assign metadata. Should be higher than ReadParallel                                                                            |
@@ -300,6 +300,29 @@ The cached files can be found in the following locations:
   * Windows: ``%APPDATA%\.fhir_packages``
   * Linux/MacOS: ``$XDG_CONFIG_HOME/.fhir_packages`` if the environment variable ``XDG_CONFIG_HOME`` is defined  otherwise ``$HOME/.config/.fhir_packages``
 
+.. _tool_fsi_recovery:
+
+Recovery Journal
+----------------
+
+If a transient error occurs while ingestion is running or the FSI instance gets interrupted, the *recovery journal* feature allows recovery from such a situation. To enable it, use the ``--useRecoveryJournal <recoveryJournalDirectory>`` option in the CLI or set field ``recoveryJournalDirectory`` in the ``appsettings.instance.config``. 
+
+When enabled, the process runs as follows:
+
+#. Upon the first ingestion attempt, FSI will take a snapshot of all the files in the specified source directory and save that snapshot to the ``<recoveryJournalDirectory>``.
+#. Then the data ingestion will start. Information about every successfully ingested resource also gets added to the journal.
+
+If the ingestion procedure gets interrupted at any point, or some of the resources do not get ingested because of a transient error (e.g. network connection to the target DB is temporarily down), the ingestion process can be restarted by running the application with the same parameters. The application will skip all the previously ingested resources based on the journal.
+
+.. note::
+
+  - Note that the recovery journal directory must be empty before performing the initial ingestion attempt for a given set of files. 
+  - Furthermore, the source files must not be changed between ingestion attempts. If any changes are detected, the FSI will throw an error.
+
+.. note::
+
+  Please do not use the source directory or any subdirectories within the source directory as the recovery journal directory.
+
 Monitoring
 ----------
 

diff --git a/features_and_tools/pubsub.rst b/features_and_tools/pubsub.rst
@@ -3,11 +3,11 @@
 Firely PubSub
 =============
 
-Firely offers the PubSub plugin to enable other services to communicate with Firely Server on data changes asynchronously. Specifically, other applications can send *commands* to update FHIR resources in the database and subscribe to *events* published by the server whenever resources change. Both commands and events get communicated as messages via a message broker (RabbitMQ).
+Firely offers the PubSub feature to enable other services to communicate with Firely Server on data changes asynchronously. Specifically, other applications can send *commands* to update FHIR resources in the database and subscribe to *events* published by the server whenever resources change. Both commands and events get communicated as messages via a message broker (RabbitMQ).
 
 Using PubSub might provide several advantages:
 
-* It is quicker than communicating via the REST API as it does not involve authorization/authentication, and resource validation. PubSub assumes that all services communicating with Firely Server are internal and secure and the resources posted using `ExecuteStorePlanCommand` are correct FHIR resources, so they do not get validated.
+* It is quicker than communicating via the REST API as it does not involve authorization/authentication and resource validation. PubSub assumes that all services communicating with Firely Server are internal and secure and that resources posted using `ExecuteStorePlanCommand` are correct FHIR resources, so they do not get validated.
 * This set-up also enables easy integration with other applications which can be written using technologies other than .NET. As long as these applications correctly implement communication via the message broker, they are able to communicate with Firely Server.
 * Having a message broker in the middle allows for building topologies where multiple applications can send commands and/or subscribe to events. 
 * If Firely Server or any of the other applications communicating with it is down, the messages will aggregate in the message broker and get processed as soon as the service is up again.
@@ -23,7 +23,7 @@ Using PubSub might provide several advantages:
 Configuration
 -------------
 
-You can enable PubSub by including the plugin in the pipeline options of the Firely Server `appsettings.instance.json` file:
+You can enable PubSub by including the plugins in the pipeline options of the Firely Server `appsettings.instance.json` file:
 
 .. code-block::
 
@@ -34,12 +34,14 @@ You can enable PubSub by including the plugin in the pipeline options of the Fir
             "Path": "/",
             "Include": [
                 ...
-                "Vonk.Plugin.PubSub"
+                "Vonk.Plugin.PubSub.Pub.MongoDb",
+                "Vonk.Plugin.PubSub.Pub.Sql",
+                "Vonk.Plugin.PubSub.Sub",
                 ]
             }
         ]
     },
-
+The ``Vonk.Plugin.PubSub.Sub`` plugin allows Firely Server to subscribe to messages published to a message broker by other clients. The ``Vonk.Plugin.PubSub.Pub.Sql`` and ``Vonk.Plugin.PubSub.Pub.MongoDb`` plugins allow Firely Server to publish changes of the respective database (either SQL or MongoDb) to the message broker. 
 You can further adjust PubSub in the PubSub section of the `appsettings.instance.json` file:
 
 .. code-block::
@@ -54,18 +56,23 @@ You can further adjust PubSub in the PubSub section of the `appsettings.instance
         },
         // The section below contains configuration related to publishing events when data gets changed in Firely Server 
         // so that other services can sync with Firely Server. Note that this is only available for Firely Server 
-        // instances that use SQL server (2016 and newer) as a repository database. 
-        // As of yet, it cannot work in combination with MongoDB or SQLite.
+        // instances that use SQL server (2016 and newer) or MongoDb as a repository database. 
+        // It does not work in combination with SQLite. 
+        // If you have configured MongoDB as a backend, note that only replica sets and sharded cluster scenarios are supported in combination with PubSub.
         "ResourceChangeNotifications": { 
             "SendLightEvents": false, // If enabled, FS will send out events on changes. These events will not contain the complete resource
             "SendFullEvents": false, // If enabled, FS will send out events on changes. These events will contain the complete resource
+            "ExcludeAuditEvents": false, // If enabled, FS will send out events on changes of resources, except Audit Events
             "PollingIntervalSeconds": 5, // How often Firely Server will be polling the DB for changes
             "MaxPublishBatchSize": 1000 // The maximum amount of resource changes that can be sent in a single message
         }
     },
 
 .. note::
-  Enabling ResourceChangeNotifications requires one-time DB configuration to enable changes tracking. See :ref:`SQL Server Tracking Initialization<pubsub_sql_tracking_init>` for the instructions.
+  Enabling ResourceChangeNotifications requires one-time DB configuration to enable changes tracking for SQL server backends. See :ref:`SQL Server Tracking Initialization<pubsub_sql_tracking_init>` for the instructions.
+
+.. note::
+  If you have configured MongoDb as your Firely Server repository database, note that the publication plugin ``Vonk.Plugin.PubSub.Pub.MongoDb`` can only be used in combination with MongoDb `replica sets <https://www.mongodb.com/docs/manual/replication/>`_ or `sharded clusters <https://www.mongodb.com/docs/manual/sharding>`_, as the plugin utilizes the :ref:`Change Stream <https://www.mongodb.com/docs/manual/changeStreams/>` functionality of MongoDb and is thus restricted.
 
 Message types and formats
 -------------------------
@@ -677,4 +684,4 @@ We provide sample code to connect to the pubsub API in the `firely-pubsub-sample
   Currently, Firely Server will setup those exchanges only once the first change in the database was detected.
   If using the `MassTransit RabbitMq nuget package <https://www.nuget.org/packages/MassTransit.RabbitMQ>`_, it will take care of setting up the exchange if not yet present.
   However, if not using this package, the client has to either take the responsibility of creating the correct exchange or wait until the exchanges are created, resulting in the loss of the first message.
-
+
diff --git a/features_and_tools/restful_api.rst b/features_and_tools/restful_api.rst
@@ -61,18 +61,14 @@ Limitations on CRUD
 Update with no changes 
 ----------------------
 
-Updating a server resource with identical content to what it currently holds results in a "No-Op" (no operation) scenario. The server acknowledges the request without making any actual modifications. To determine if your action resulted in a No-Op, you can check the OperationOutcome resource provided by the server. To prompt the server for this outcome, you might have to configure the Prefer Header. Following the FHIR specification, the server does not automatically provide an outcome response by default. For more details, see the `FHIR documentation <https://build.fhir.org/http.html#ops>`_.
-When the Update No-Op plugin is enabled:
-- It reduces server load and data processing by avoiding redundant updates.
-- It helps in controlling database growth by preventing the creation of unnecessary new versions of resources.
+Updating Firely Server with a resource that is identical to an existing resource in the database will normally create a new version of that resource, along with new AuditEvent and Provenance resources if auditing is enabled.
+This can put extra load on the server where this is not entirely necessary. To avoid these updates that can unnecessarily increase server load and database growth, the No-Op (No Operation) plugins can be enabled.
+By enabling these plugins, the server acknowledges the request without making any actual modifications to the database. If an update resulted in a No-Op scenario, this can be observed in the OperationOutcome that is returned by Firely Server.
 
 Configuration for No-Op
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-Update with no changes requires `UpdateNoOp` plugin to be enabled. 
-The plugin can be configured to ignore additional meta elements in the resource. 
-By default, when plugin is enabled the following meta elements are ignored during resource comparison: ``versionId``, ``lastUpdated`` and ``source`` You can add additional meta elements to be ignored
-You can also add ``security``, ``tag`` and ``profile`` or any other to be ignored, but it depends on specific usage of meta. See more on `FHIR article <https://www.hl7.org/fhir/resource.html#tag-updates>`_.
+To make sure Firely server uses the No-Op scenario, the `UpdateNoOp` plugins need to be enabled in the `PipelineOptions`. 
 ::
    "PipelineOptions": {
    "PluginDirectory": "./plugins",
@@ -95,6 +91,51 @@ You can also add ``security``, ``tag`` and ``profile`` or any other to be ignore
       ]
    }
 
+There are three No-Op plugins available:
+
+* ``Vonk.Plugin.UpdateNoOp.UpdateNoOpConfiguration`` - For regular updates
+* ``Vonk.Plugin.UpdateNoOp.PatchNoOpConfiguration`` - For Patch operations
+* ``Vonk.Plugin.UpdateNoOp.ConditionalUpdateNoOpConfiguration`` - For conditional updates
+
+By default the following meta elements are ignored during resource comparison: ``versionId``, ``lastUpdated`` and ``source``. You can also add ``security``, ``tag`` and ``profile`` or any other meta element to be ignored, but it depends on your specific usage of meta. For more information see `the hl7 specification <https://www.hl7.org/fhir/resource.html#tag-updates>`_.
+
+To determine if your action resulted in a No-Op scenario, you can configure Firely Server to return an OperationOutcome. For this it is necessary to configure the Prefer Header as Firely Server does not return this response by default.
+The Prefer Header can be set in three ways, as per `the hl7 specification <https://build.fhir.org/http.html#ops>`_:
+
+* ``return=minimal``- Nothing is returned by the server
+* ``return=representation`` - The resource is returned as present in the database
+* ``return=OperationOutcome`` - Return an OperationOutcome
+
+In the example below an OperationOutcome for a No-Op scenario is returned when the Prefer Header is set to ``return=OperationOutcome``:
+::
+   {
+    "resourceType": "OperationOutcome",
+    "id": "26a724d9-10e4-4a71-819e-15d52f6f821c",
+    "meta": {
+        "versionId": "b6063533-a93e-4cd1-bb0b-5f37381d0f20",
+        "lastUpdated": "2024-02-12T11:12:40.6172822+00:00"
+    },
+    "issue": [
+        {
+            "severity": "information",
+            "code": "informational",
+            "details": {
+                "coding": [
+                    {
+                        "system": "http://hl7.org/fhir/dotnet-api-operation-outcome",
+                        "code": "5025"
+                    }
+                ],
+                "text": "No changes were performed as the provided resource contains no changes to the existing resource"
+            }
+        },
+        {
+            "severity": "information",
+            "code": "informational",
+            "diagnostics": "The operation was successful"
+        }
+    ]
+   }
 
 .. _restful_versioning:
 

diff --git a/maintenance/performance_firely_server.rst b/maintenance/performance_firely_server.rst
@@ -35,7 +35,7 @@ When a resource is sent to Firely Server for storage, Firely Server indexes the
 
 Additionally, these search parameters can have an impact on the search performance:
 
-#. ``_total``: Searchbundles contain the element ``total`` which indicates the total number of resources that match the query's search parameters. Setting this parameter to ``_total = none`` results in faster searches as it saves the query that would generate the result of the aforementioned element.
+#. ``_total``: Searchbundles contain the element ``total`` which indicates the total number of resources that match the query's search parameters. Setting this parameter to ``_total = none`` results in faster searches as it saves the query that would generate the result of the aforementioned element. Default value can be changed for `_total` in the configuration, see :ref:`bundle_options`.
 
 
 Pipeline

diff --git a/reference/dependencies.rst b/reference/dependencies.rst
@@ -25,7 +25,6 @@ This list uses the NuGet package names (or prefixes of them) so you can easily l
 #. IdentityModel.* - Apache 2.0
 #. IdentityServer4.AccessTokenValidation - Apache 2.0
 #. IPNetwork2 - BSD 2-Clause "Simplified" License
-#. Quartz - Apache 2.0
 #. Serilog(.*) - Apache-2.0
 #. LinqKit.Microsoft.EntityFrameworkCore - MIT
 #. Hl7.Fhir.* - Firely OSS license (see below)