Merge branch 'develop' into 7457-introduce-mpconfig

doc/release-notes/7418-datasourcedefinition.md

@@ -0,0 +1,88 @@
+## Release Highlights
+
+### Easier Configuration of Database Connections
+
+Dataverse now being able to use up-to-date Java technologies, transforms
+the way how to configure the connection to your PostgreSQL database.
+
+In the past, the configuration of the connection has been quite static
+and not very easy to update. This has been an issue especially for cloud
+and container usage.
+
+Using MicroProfile Config API (#7000, #7418), you can much more easily specify configuration
+details. For an overview of supported options, please see the
+[installation guide](https://guides.dataverse.org/en/5.3/installation/config.html#jvm-options).
+
+Note that some settings have been moved from domain.xml to code such as min and max pool size.
+
+## Notes for Dataverse Installation Administrators
+
+### New JVM Options
+
+- dataverse.db.name
+- dataverse.db.user
+- dataverse.db.password
+- dataverse.db.host
+- dataverse.db.port
+
+<!-- ## Update to Payara Platform 5.2020.6 -->
+<!-- ... -->
+
+<!-- PLACEHOLDER REPLACEMENT TEXT FOR PAYARA UPGRADE NOTE #7417 -->
+🚨 THIS VERSION OF DATAVERSE **REQUIRES** UPGRADING TO PAYARA 5.2020.6. 🚨
+
+<!-- ... -->
+
+## Upgrading from earlier releases
+
+ℹ️ You need to update the Payara Application Server before continuing here. See above.
+
+1. Undeploy the previous version.
+```
+<payara install path>/asadmin list-applications
+<payara install path>/asadmin undeploy dataverse-<version>
+```
+
+(where `<payara install path>` is where Payara 5 is installed, for example: `/usr/local/payara5`)
+
+2. Update your database connection before updating.
+
+Please configure your connection details, replacing all the `${DB_...}`.
+(If you are using a PostgreSQL server on `localhost:5432`, you can omit `dataverse.db.host` and `dataverse.db.port`.)
+
+```
+<payara install path>/asadmin create-system-properties "dataverse.db.user=${DB_USER}"
+<payara install path>/asadmin create-system-properties "dataverse.db.host=${DB_HOST}"
+<payara install path>/asadmin create-system-properties "dataverse.db.port=${DB_PORT}"
+<payara install path>/asadmin create-system-properties "dataverse.db.name=${DB_NAME}"
+echo "AS_ADMIN_ALIASPASSWORD=${DB_PASS}" > /tmp/password.txt
+<payara install path>/asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password
+rm /tmp/password.txt
+```
+
+<!-- PLACE HOLDER FOR EJB TIMER DATABASE RESET NOTE #5345 -->
+
+Now you are safe to delete the old password alias and DB pool:
+```
+<payara install path>/asadmin delete-jdbc-connection-pool --cascade=true dvnDbPool
+<payara install path>/asadmin delete-password-alias db_password_alias
+```
+
+3. Stop payara and remove the generated directory, start.
+```
+service payara stop
+# remove the generated directory:
+rm -rf <payara install path>/payara/domains/domain1/generated
+service payara start
+```
+
+3. Deploy this version.
+```
+<payara install path>/bin/asadmin deploy dataverse-5.3.war
+```
+
+4. Restart Payara
+```
+service payara stop
+service payara start
+```

doc/release-notes/7423-jms-move-to-code.md

@@ -0,0 +1,13 @@
+# Java Message System Configuration
+
+The ingest part of Dataverse uses the Java Message System to create ingest tasks in a queue.
+That queue had been configured from command line or domain.xml before. This has now changed to being done
+in code.
+
+In the unlikely case you might want to change any of these settings, feel free to change and recompile or raise an issue.
+See `IngestQueueProducer` for more details.
+
+If you want to clean up your existing installation, you can delete the old, unused queue like this:
+```shell
+asadmin delete-connector-connection-pool --cascade=true jms/IngestQueueConnectionFactoryPool
+```

doc/sphinx-guides/source/installation/advanced.rst

@@ -17,7 +17,7 @@ You should be conscious of the following when running multiple app servers.
 - When a sitemap is created by an app server it is written to the filesystem of just that app server. By default the sitemap is written to the directory ``/usr/local/payara5/glassfish/domains/domain1/docroot/sitemap``.
 - If Make Data Count is used, its raw logs must be copied from each app server to single instance of Counter Processor. See also :ref:`:MDCLogPath` section in the Configuration section of this guide and the :doc:`/admin/make-data-count` section of the Admin Guide.
 - Dataset draft version logging occurs separately on each app server. See :ref:`edit-draft-versions-logging` section in Monitoring of the Admin Guide for details.
-- Password aliases (``db_password_alias``, etc.) are stored per app server.
+- Password aliases (``dataverse.db.password``, etc.) are stored per app server.
 
 Detecting Which App Server a User Is On
 +++++++++++++++++++++++++++++++++++++++

doc/sphinx-guides/source/installation/config.rst

@@ -1032,6 +1032,57 @@ dataverse.auth.password-reset-timeout-in-minutes
 
 Users have 60 minutes to change their passwords by default. You can adjust this value here.
 
+dataverse.db.name
++++++++++++++++++
+
+The PostgreSQL database name to use for Dataverse.
+
+Defaults to ``dataverse`` (but the installer sets it to ``dvndb``).
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_NAME``.
+
+dataverse.db.user
++++++++++++++++++
+
+The PostgreSQL user name to connect with.
+
+Defaults to ``dataverse`` (but the installer sets it to ``dvnapp``).
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_USER``.
+
+dataverse.db.password
++++++++++++++++++++++
+
+The PostgreSQL users password to connect with.
+
+Preferrably use a JVM alias, as passwords in environment variables aren't safe.
+
+.. code-block:: shell
+
+  echo "AS_ADMIN_ALIASPASSWORD=change-me-super-secret" > /tmp/password.txt
+  asadmin create-password-alias --passwordfile /tmp/password.txt dataverse.db.password
+  rm /tmp/password.txt
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_PASSWORD``.
+
+dataverse.db.host
++++++++++++++++++
+
+The PostgreSQL server to connect to.
+
+Defaults to ``localhost``.
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_HOST``.
+
+dataverse.db.port
++++++++++++++++++
+
+The PostgreSQL server port to connect to.
+
+Defaults to ``5432``, the default PostgreSQL port.
+
+Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_PORT``.
+
 dataverse.rserve.host
 +++++++++++++++++++++
 
@@ -1168,7 +1219,7 @@ This JVM setting is also part of **handles** configuration. The Handle.Net insta
 .. _dataverse.handlenet.index:
 
 dataverse.handlenet.index
-+++++++++++++++++++++++++++++++++
++++++++++++++++++++++++++
 If you want to use different index than the default 300
 
 .. _dataverse.timerServer:

doc/sphinx-guides/source/installation/installation-main.rst

@@ -74,9 +74,9 @@ All the Payara configuration tasks performed by the installer are isolated in th
 
 While Postgres can accomodate usernames and database names containing hyphens, it is strongly recommended to use only alphanumeric characters.
 
-**IMPORTANT:** As a security measure, the ``as-setup.sh`` script stores passwords as "aliases" rather than plaintext. If you change your database password, for example, you will need to update the alias with ``asadmin update-password-alias db_password_alias``, for example. Here is a list of the password aliases that are set by the installation process and entered into Payara's ``domain.xml`` file:
+**IMPORTANT:** As a security measure, the ``as-setup.sh`` script stores passwords as "aliases" rather than plaintext. If you change your database password, for example, you will need to update the alias with ``asadmin update-password-alias dataverse.db.password``, for example. Here is a list of the password aliases that are set by the installation process and entered into Payara's ``domain.xml`` file:
 
-- ``db_password_alias``
+- ``dataverse.db.password``
 - ``doi_password_alias``
 - ``rserve_password_alias``
 

pom.xml

@@ -696,6 +696,7 @@
                     <include>**/*.xml</include>
                     <include>**/firstNames/*.*</include>
                     <include>**/*.xsl</include>
+                    <include>**/*.properties</include>
                 </includes>
             </resource>
         </resources>

scripts/installer/as-setup.sh

@@ -73,7 +73,7 @@ function preliminary_setup()
   ./asadmin $ASADMIN_OPTS create-jvm-options "-XX\:+DisableExplicitGC"
 
   # alias passwords
-  for alias in "rserve_password_alias ${RSERVE_PASS}" "doi_password_alias ${DOI_PASSWORD}" "db_password_alias ${DB_PASS}"
+  for alias in "rserve_password_alias ${RSERVE_PASS}" "doi_password_alias ${DOI_PASSWORD}" "dataverse.db.password ${DB_PASS}"
   do
       set -- $alias
       echo "AS_ADMIN_ALIASPASSWORD=$2" > /tmp/$1.txt
@@ -122,26 +122,6 @@ function preliminary_setup()
   # bump the http-listener timeout from 900 to 3600
   ./asadmin $ASADMIN_OPTS set server-config.network-config.protocols.protocol.http-listener-1.http.request-timeout-seconds="${GLASSFISH_REQUEST_TIMEOUT}"
 
-  ./asadmin $ASADMIN_OPTS delete-connector-connection-pool --cascade=true jms/__defaultConnectionFactory-Connection-Pool 
-
-  # no need to explicitly delete the connector resource for the connection pool deleted in the step 
-  # above - the cascade delete takes care of it.
-  #./asadmin $ASADMIN_OPTS delete-connector-resource jms/__defaultConnectionFactory-Connection-Pool
-
-  # http://docs.oracle.com/cd/E19798-01/821-1751/gioce/index.html
-  ./asadmin $ASADMIN_OPTS create-connector-connection-pool --steadypoolsize 1 --maxpoolsize 250 --poolresize 2 --maxwait 60000 --raname jmsra --connectiondefinition javax.jms.QueueConnectionFactory jms/IngestQueueConnectionFactoryPool
-
-  # http://docs.oracle.com/cd/E18930_01/html/821-2416/abllx.html#giogt
-  ./asadmin $ASADMIN_OPTS create-connector-resource --poolname jms/IngestQueueConnectionFactoryPool --description "ingest connector resource" jms/IngestQueueConnectionFactory
-
-  # http://docs.oracle.com/cd/E18930_01/html/821-2416/ablmc.html#giolr
-  ./asadmin $ASADMIN_OPTS create-admin-object --restype javax.jms.Queue --raname jmsra --description "sample administered object" --property Name=DataverseIngest jms/DataverseIngest
-
-  # no need to explicitly create the resource reference for the connection factory created above -
-  # the "create-connector-resource" creates the reference automatically.
-  #./asadmin $ASADMIN_OPTS create-resource-ref --target Cluster1 jms/IngestQueueConnectionFactory
-
-
   # so we can front with apache httpd ( ProxyPass / ajp://localhost:8009/ )
   ./asadmin $ASADMIN_OPTS create-network-listener --protocol http-listener-1 --listenerport 8009 --jkenabled true jk-connector
 }
@@ -150,17 +130,11 @@ function final_setup(){
         ./asadmin $ASADMIN_OPTS delete-jvm-options -Xmx512m
         ./asadmin $ASADMIN_OPTS create-jvm-options "-Xmx${MEM_HEAP_SIZE}m"
 
-
-        ./asadmin $ASADMIN_OPTS create-jdbc-connection-pool --restype javax.sql.DataSource \
-                                        --datasourceclassname org.postgresql.ds.PGPoolingDataSource \
-                                        --property create=true:User=$DB_USER:PortNumber=$DB_PORT:databaseName=$DB_NAME:ServerName=$DB_HOST \
-                                        dvnDbPool
-
-       ./asadmin $ASADMIN_OPTS set resources.jdbc-connection-pool.dvnDbPool.property.password='${ALIAS=db_password_alias}'
-
-        ###
-        # Create data sources
-        ./asadmin $ASADMIN_OPTS create-jdbc-resource --connectionpoolid dvnDbPool jdbc/VDCNetDS
+         # Set up the database connection properties
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.user=${DB_USER}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.host=${DB_HOST}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.port=${DB_PORT}"
+        ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.db.name=${DB_NAME}"
 
         ./asadmin $ASADMIN_OPTS create-jvm-options "\-Djavax.xml.parsers.SAXParserFactory=com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl"
 

src/main/java/edu/harvard/iq/dataverse/flyway/StartupFlywayMigrator.java

@@ -15,7 +15,7 @@
 @TransactionManagement(value = TransactionManagementType.BEAN)
 public class StartupFlywayMigrator {
 
-    @Resource(lookup = "jdbc/VDCNetDS")
+    @Resource(lookup = "java:app/jdbc/dataverse")
     private DataSource dataSource;
 
     @PostConstruct

src/main/java/edu/harvard/iq/dataverse/ingest/IngestMessageBean.java

@@ -45,7 +45,13 @@
  * 
  * @author Leonid Andreev 
  */
-@MessageDriven(mappedName = "jms/DataverseIngest", activationConfig =  {@ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"), @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")})
+@MessageDriven(
+    mappedName = "java:app/jms/queue/ingest",
+    activationConfig =  {
+        @ActivationConfigProperty(propertyName = "acknowledgeMode", propertyValue = "Auto-acknowledge"),
+        @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue")
+    }
+)
 public class IngestMessageBean implements MessageListener {
     private static final Logger logger = Logger.getLogger(IngestMessageBean.class.getCanonicalName());
     @EJB DatasetServiceBean datasetService;

src/main/java/edu/harvard/iq/dataverse/ingest/IngestQueueProducer.java

@@ -0,0 +1,50 @@
+package edu.harvard.iq.dataverse.ingest;
+
+import javax.annotation.Resource;
+// https://www.baeldung.com/jee-cdi-vs-ejb-singleton
+import javax.inject.Singleton;
+import javax.enterprise.inject.Produces;
+import javax.jms.JMSConnectionFactoryDefinition;
+import javax.jms.JMSDestinationDefinition;
+import javax.jms.Queue;
+import javax.jms.QueueConnectionFactory;
+
+@JMSConnectionFactoryDefinition(
+    description = "Dataverse Ingest Queue Factory",
+    name = "java:app/jms/factory/ingest",
+    resourceAdapter = "jmsra",
+    interfaceName = "javax.jms.QueueConnectionFactory",
+    maxPoolSize = 250,
+    minPoolSize = 1,
+    properties = {
+        "org.glassfish.connector-connection-pool.max-wait-time-in-millis=60000",
+        "org.glassfish.connector-connection-pool.pool-resize-quantity=2"
+    }
+)
+@JMSDestinationDefinition(
+    description = "Dataverse Ingest Queue",
+    name = "java:app/jms/queue/ingest",
+    resourceAdapter = "jmsra",
+    interfaceName="javax.jms.Queue",
+    destinationName = "DataverseIngest"
+)
+@Singleton
+public class IngestQueueProducer {
+
+    @Resource(lookup="java:app/jms/queue/ingest")
+    Queue ingestQueue;
+
+    @Resource(lookup="java:app/jms/factory/ingest")
+    QueueConnectionFactory ingestQueueFactory;
+
+    @Produces
+    public Queue getIngestQueue() {
+        return ingestQueue;
+    }
+
+    @Produces
+    public QueueConnectionFactory getIngestQueueFactory() {
+        return ingestQueueFactory;
+    }
+
+}

src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java

@@ -136,9 +136,9 @@ public class IngestServiceBean {
     @EJB
     SystemConfig systemConfig;
 
-    @Resource(mappedName = "jms/DataverseIngest")
+    @Resource(name = "java:app/jms/queue/ingest")
     Queue queue;
-    @Resource(mappedName = "jms/IngestQueueConnectionFactory")
+    @Resource(name = "java:app/jms/factory/ingest")
     QueueConnectionFactory factory;