Merge pull request #7463 from poikilotherm/7457-introduce-mpconfig

7457 introduce mpconfig

doc/release-notes/7457-introduce-mpconfig.md

@@ -0,0 +1,10 @@
+## Introducing MicroProfile Config API
+
+With this Dataverse release, we start to make use of the MicroProfile Config API. (As you might have noticed
+for the database connection settings.)
+
+This will benefit both devs and sysadmins, but the codebase will have to be refactored to make use of it.
+As this will take time, we will always provide a backward compatible way of using it.
+
+For more details, please see the development guide about "Consuming Configuration", which also
+explains the benefits in more detail.

doc/sphinx-guides/source/developers/configuration.rst

@@ -0,0 +1,100 @@
+Consuming Configuration
+=======================
+
+.. contents:: |toctitle|
+	:local:
+
+Dataverse uses different types of configuration:
+
+1. JVM system properties
+2. Simple database value settings
+3. Complex database stored data structures
+
+1 and 2 are usually simple text strings, boolean switches or digits. All of those can be found in :doc:`/installation/config`.
+
+Anything for 3 is configured via the API using either TSV or JSON structures. Examples are metadata blocks,
+authentication providers, harvesters and others.
+
+Simple Configuration Options
+----------------------------
+
+Developers have accessed the simple properties via
+
+1. ``System.getProperty(...)`` for JVM system property settings
+2. ``SettingsServiceBean.get(...)`` for database settings and
+3. ``SystemConfig.xxx()`` for specially treated settings, maybe mixed from 1 and 2 and other sources.
+4. ``SettingsWrapper``, reading from 2 and 3 for use in frontend pages based on JSF
+
+As of Dataverse 5.3, we start to streamline our efforts into using a more consistent approach, also bringing joy and
+happiness to all the system administrators out there. This will be done by adopting the use of
+`MicroProfile Config <https://github.com/eclipse/microprofile-config>`_ over time.
+
+So far we streamlined configuration of these Dataverse parts:
+
+- ✅ Database Connection
+
+Complex Configuration Options
+-----------------------------
+
+We should enable variable substitution in JSON configuration. Example: using substitution to retrieve values from
+MicroProfile Config and insert into the authentication provider would allow much easier provisioning of secrets
+into the providers.
+
+Why should I care about MicroProfile Config API?
+------------------------------------------------
+
+Developers benefit from:
+
+- A streamlined API to retrieve configuration, backward-compatible renaming strategies and easier testbed configurations.
+- Config API is also pushing for validation of configuration, as it's typesafe and converters for non-standard types
+  can be added within our codebase.
+- Defaults in code or bundled in ``META-INF/microprofile-config.properties`` allow for optional values without much hassle.
+
+System administrators benefit from:
+
+- Lots of database settings have been introduced in the past, but should be more easily configurable and not rely on a
+  database connection.
+- Running Dataverse in containers gets much easier when configuration can be provisioned in a
+  streamlined fashion, mitigating the need for scripting glue and distinguishing between setting types.
+- Classic installations have a profit, too: we can enable using a single config file, e.g. living in
+  ``/etc/dataverse/config.properties``.
+- Features for monitoring resources and others are easier to use with this streamlined configuration, as we can
+  avoid people having to deal with ``asadmin`` commands and change a setting comfortably instead.
+
+Adopting MicroProfile Config API
+---------------------------------
+
+This technology is introduced on a step-by-step basis. There will not be a big shot, crashing upgrades for everyone.
+Instead, we will provide backward compatibility by deprecating renamed or moved config options, while still
+supporting the old way of setting them.
+
+- Introducing a new setting or moving and old one should result in a key ``dataverse.<scope/task/module/...>.<setting>``.
+  That way we enable sys admins to recognize the meaning of an option and avoid name conflicts.
+  Starting with ``dataverse`` makes it perfectly clear that this is a setting meant for this application, which is
+  important when using environment variables, system properties or other MPCONFIG sources.
+- Replace ``System.getProperty()`` calls with either injected configs or retrieve programmatically if more complex
+  handling is necessary. If you rename the property, you should provide an alias. See below.
+- Database settings need to be refactored in multiple steps. First you need to change the code retrieving it to use
+  MicroProfile Config API instead (just like above). Then you should provide an alias to retain backward compatibility.
+  See below.
+
+Moving or Replacing a JVM Setting
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When moving an old key to a new (especially when doing so with a former JVM system property setting), you should
+add an alias to ``src/main/resources/META-INF/microprofile-aliases.properties`` to enable backward compatibility.
+The format is always like ``dataverse.<scope/....>.newname...=old.property.name``.
+
+Details can be found in ``edu.harvard.iq.dataverse.settings.source.AliasConfigSource``
+
+Aliasing Database Setting
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+When moving a database setting (``:ExampleSetting``), configure an alias
+``dataverse.my.example.setting=dataverse.settings.fromdb.ExampleSetting`` in
+``src/main/resources/META-INF/microprofile-aliases.properties``. This will enable backward compatibility.
+
+A database setting with an i18n attribute using *lang* will have available language codes appended to the name.
+Example: ``dataverse.settings.fromdb.ExampleI18nSetting.en``, ``dataverse.settings.fromdb.ExampleI18nSetting.de``
+
+More details in ``edu.harvard.iq.dataverse.settings.source.DbSettingConfigSource``

doc/sphinx-guides/source/developers/index.rst

@@ -22,6 +22,7 @@ Developer Guide
    dependencies
    debugging
    coding-style
+   configuration
    deployment
    containers
    making-releases

doc/sphinx-guides/source/developers/intro.rst

@@ -28,6 +28,8 @@ Dataverse is a `Jakarta EE <https://en.wikipedia.org/wiki/Jakarta_EE>`_ applicat
 
 We make use of a variety of Jakarta EE technologies such as JPA, JAX-RS, JMS, and JSF. The front end is built using PrimeFaces and Bootstrap.
 
+In addition, we start to adopt parts of Eclipse MicroProfile, namely `MicroProfile Config <https://github.com/eclipse/microprofile-config>`_.
+
 Roadmap
 -------
 

pom.xml

@@ -34,7 +34,8 @@
         <junit.version>4.13.1</junit.version>
         <junit.jupiter.version>5.7.0</junit.jupiter.version>
         <junit.vintage.version>${junit.jupiter.version}</junit.vintage.version>
-        <testcontainers.version>1.13.0</testcontainers.version>
+        <testcontainers.version>1.15.0</testcontainers.version>
+        <microbean-mpconfig.version>0.4.1</microbean-mpconfig.version>
         <mockito.version>2.28.2</mockito.version>
         <flyway.version>5.2.4</flyway.version>
         <jhove.version>1.20.1</jhove.version>
@@ -280,6 +281,11 @@
             <version>29.0-jre</version>
             <type>jar</type>
         </dependency>
+        <dependency>
+            <groupId>org.eclipse.microprofile.config</groupId>
+            <artifactId>microprofile-config-api</artifactId>
+            <scope>provided</scope>
+        </dependency>
         <dependency>
             <groupId>jakarta.platform</groupId>
             <artifactId>jakarta.jakartaee-api</artifactId>
@@ -637,6 +643,11 @@
             <artifactId>junit-jupiter</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>postgresql</artifactId>
+            <scope>test</scope>
+        </dependency>
         <dependency>
             <groupId>org.mockito</groupId>
             <artifactId>mockito-core</artifactId>
@@ -649,7 +660,12 @@
             <version>${mockito.version}</version>
             <scope>test</scope>
         </dependency>
-
+        <dependency>
+            <groupId>org.microbean</groupId>
+            <artifactId>microbean-microprofile-config</artifactId>
+            <version>${microbean-mpconfig.version}</version>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
     <build>
         <!--        <testResources>
@@ -686,6 +702,7 @@
                     <include>**/firstNames/*.*</include>
                     <include>**/*.xsl</include>
                     <include>**/*.properties</include>
+                    <include>**/services/*</include>
                 </includes>
             </resource>
         </resources>
@@ -844,6 +861,7 @@
             <id>tc</id>
             <properties>
                 <skipUnitTests>true</skipUnitTests>
+                <postgresql.server.version>9.6</postgresql.server.version>
             </properties>
             <build>
                 <plugins>
@@ -853,6 +871,9 @@
                         <version>2.22.2</version>
                         <configuration>
                             <groups>testcontainers</groups>
+                            <systemPropertyVariables>
+                                <postgresql.server.version>${postgresql.server.version}</postgresql.server.version>
+                            </systemPropertyVariables>
                         </configuration>
                         <executions>
                             <execution>

src/main/java/edu/harvard/iq/dataverse/settings/source/AliasConfigSource.java

@@ -0,0 +1,98 @@
+package edu.harvard.iq.dataverse.settings.source;
+
+import org.eclipse.microprofile.config.ConfigProvider;
+import org.eclipse.microprofile.config.spi.ConfigSource;
+
+import java.io.IOException;
+import java.net.URL;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import java.util.logging.Logger;
+
+/**
+ * Enable using an old name for a new config name.
+ * Usages will be logged and this source will ALWAYS stand back if the new name is used anywhere.
+ *
+ * By using a DbSettingConfigSource value (dataverse.settings.fromdb.XXX) as old name, we can
+ * alias a new name to an old db setting, enabling backward compatibility.
+ */
+public final class AliasConfigSource implements ConfigSource {
+
+    private static final Logger logger = Logger.getLogger(AliasConfigSource.class.getName());
+
+    private final ConcurrentHashMap<String, String> aliases = new ConcurrentHashMap<>();
+    private final String ALIASES_PROP_FILE = "META-INF/microprofile-aliases.properties";
+
+    public AliasConfigSource() {
+        try {
+            // read properties from class path file
+            Properties aliasProps = readAliases(ALIASES_PROP_FILE);
+            // store in our aliases map
+            importAliases(aliasProps);
+        } catch (IOException e) {
+            logger.info("Could not read from "+ALIASES_PROP_FILE+". Skipping MPCONFIG alias setup.");
+        }
+    }
+
+    Properties readAliases(String filePath) throws IOException {
+        // get resource from classpath
+        ClassLoader classLoader = this.getClass().getClassLoader();
+        URL aliasesResource = classLoader.getResource(filePath);
+
+        // load properties from file resource (parsing included)
+        Properties aliasProps = new Properties();
+        try {
+            aliasProps.load(aliasesResource.openStream());
+        } catch (NullPointerException e) {
+            throw new IOException(e.getMessage());
+        }
+        return aliasProps;
+    }
+
+    void importAliases(Properties aliasProps) {
+        aliasProps.forEach((key, value) -> aliases.put(key.toString(), value.toString()));
+    }
+
+    @Override
+    public Map<String, String> getProperties() {
+        // No, just no. We're not going to drop a list of stuff. We're only
+        // dealiasing on getValue();
+        return new HashMap<>();
+    }
+
+    @Override
+    public Set<String> getPropertyNames() {
+        // It does not make sense to retrieve the aliased names...
+        return new HashSet<>();
+    }
+
+    @Override
+    public int getOrdinal() {
+        // Any other config source can override us.
+        // As soon as someone is starting to use the new property name, this alias becomes pointless.
+        return Integer.MIN_VALUE;
+    }
+
+    @Override
+    public String getValue(String key) {
+        String value = null;
+        if (this.aliases.containsKey(key)) {
+            String oldKey = this.aliases.get(key);
+            value = ConfigProvider.getConfig().getOptionalValue(oldKey, String.class).orElse(null);
+
+            if (value != null) {
+                logger.warning("Detected deprecated config option '"+oldKey+"' in use. Please update your config to use '"+key+"'.");
+            }
+        }
+        return value;
+    }
+
+    @Override
+    public String getName() {
+        return "Alias";
+    }
+}

src/main/java/edu/harvard/iq/dataverse/settings/source/DbSettingConfigHelper.java

@@ -0,0 +1,27 @@
+package edu.harvard.iq.dataverse.settings.source;
+
+import edu.harvard.iq.dataverse.settings.SettingsServiceBean;
+
+import javax.annotation.PostConstruct;
+import javax.ejb.EJB;
+import javax.ejb.Singleton;
+import javax.ejb.Startup;
+
+/**
+ * This is a small helper bean for the MPCONFIG DbSettingConfigSource.
+ * As it is a singleton and built at application start (=deployment), it will inject the (stateless)
+ * settings service into the MPCONFIG POJO once it's ready.
+ *
+ * MPCONFIG requires it's sources to be POJOs. No direct dependency injection possible.
+ */
+@Singleton
+@Startup
+public class DbSettingConfigHelper {
+    @EJB
+    SettingsServiceBean settingsSvc;
+
+    @PostConstruct
+    public void injectService() {
+        DbSettingConfigSource.injectSettingsService(settingsSvc);
+    }
+}