Merge pull request #9939 from poikilotherm/7424-maildefinition-ng

Enable mail MTA configuration using MicroProfile Config

doc/release-notes/7424-mailsession.md

@@ -0,0 +1,24 @@
+## Simplified SMTP configuration
+
+With this release, we deprecate the usage of `asadmin create-javamail-resource` to configure Dataverse to send mail using your SMTP server and provide a simplified, standard alternative using JVM options or MicroProfile Config.
+
+At this point, no action is required if you want to keep your current configuration.
+Warnings will show in your server logs to inform and remind you about the deprecation.
+A future major release of Dataverse may remove this way of configuration.
+
+Please do take the opportunity to update your SMTP configuration. Details can be found in section of the Installation Guide starting with the [SMTP/Email Configuration](https://guides.dataverse.org/en/6.2/installation/config.html#smtp-email-configuration) section of the Installation Guide.
+
+Once reconfiguration is complete, you should remove legacy, unused config. First, run `asadmin delete-javamail-resource mail/notifyMailSession` as described in the [6.1 guides](https://guides.dataverse.org/en/6.1/installation/installation-main.html#mail-host-configuration-authentication). Then run `curl -X DELETE http://localhost:8080/api/admin/settings/:SystemEmail` as this database setting has been replace with `dataverse.mail.system-email` as described below.
+
+Please note: as there have been problems with email delivered to SPAM folders when the "From" within mail envelope and the mail session configuration didn't match (#4210), as of this version the sole source for the "From" address is the setting `dataverse.mail.system-email` once you migrate to the new way of configuration. 
+
+List of options added:
+- dataverse.mail.system-email
+- dataverse.mail.mta.host
+- dataverse.mail.mta.port
+- dataverse.mail.mta.ssl.enable
+- dataverse.mail.mta.auth
+- dataverse.mail.mta.user
+- dataverse.mail.mta.password
+- dataverse.mail.mta.allow-utf8-addresses
+- Plus many more for advanced usage and special provider requirements. See [configuration guide for a full list](https://guides.dataverse.org/en/6.2/installation/config.html#dataverse-mail-mta).

doc/sphinx-guides/source/container/app-image.rst

@@ -134,19 +134,6 @@ In addition, the application image provides the following tunables:
 
         1. Simply pick a JVM option from the list and replace any ``.`` with ``_``.
         2. Replace any ``-`` in the option name with ``__``.
-    * - ``DATAVERSE_MAIL_HOST``
-      - ``smtp``
-      - String
-      - A hostname (w/o port!) where to reach a Mail MTA on port 25.
-    * - ``DATAVERSE_MAIL_USER``
-      - ``dataversenotify``
-      - String
-      - A username to use with the Mail MTA
-    * - ``DATAVERSE_MAIL_FROM``
-      - ``dataverse@localhost``
-      - Mail address
-      - The "From" field for all outbound mail. Make sure to set :ref:`systemEmail` to the same value or no mail will
-        be sent.
 
 
 Note that the script ``init_2_configure.sh`` will apply a few very important defaults to enable quick usage

doc/sphinx-guides/source/installation/installation-main.rst

@@ -141,66 +141,6 @@ Got ERR_ADDRESS_UNREACHABLE While Navigating on Interface or API Calls
 
 If you are receiving an ``ERR_ADDRESS_UNREACHABLE`` while navigating the GUI or making an API call, make sure the ``siteUrl`` JVM option is defined. For details on how to set ``siteUrl``, please refer to :ref:`dataverse.siteUrl` from the :doc:`config` section. For context on why setting this option is necessary, refer to :ref:`dataverse.fqdn` from the :doc:`config` section.
 
-Problems Sending Email
-^^^^^^^^^^^^^^^^^^^^^^
-
-If your Dataverse installation is not sending system emails, you may need to provide authentication for your mail host. First, double check the SMTP server being used with this Payara asadmin command:
-
-``./asadmin get server.resources.mail-resource.mail/notifyMailSession.host``
-
-This should return the DNS of the mail host you configured during or after installation. mail/notifyMailSession is the JavaMail Session that's used to send emails to users. 
-
-If the command returns a host you don't want to use, you can modify your notifyMailSession with the Payara ``asadmin set`` command with necessary options (`click here for the manual page <https://docs.oracle.com/cd/E18930_01/html/821-2433/set-1.html>`_), or via the admin console at http://localhost:4848 with your domain running. 
-
-If your mail host requires a username/password for access, continue to the next section.
-
-Mail Host Configuration & Authentication
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-If you need to alter your mail host address, user, or provide a password to connect with, these settings are easily changed in the Payara admin console or via command line. 
-
-For the Payara console, load a browser with your domain online, navigate to http://localhost:4848 and on the side panel find JavaMail Sessions. By default, the Dataverse Software uses a session named mail/notifyMailSession for routing outgoing emails. Click this mail session in the window to modify it.
-
-When fine tuning your JavaMail Session, there are a number of fields you can edit. The most important are:
-
-+ **Mail Host:** Desired mail host’s DNS address (e.g. smtp.gmail.com)
-+ **Default User:** Username mail host will recognize (e.g. user\@gmail.com)
-+ **Default Sender Address:** Email address that your Dataverse installation will send mail from
-
-Depending on the SMTP server you're using, you may need to add additional properties at the bottom of the page (below "Advanced").
-
-From the "Add Properties" utility at the bottom, use the “Add Property” button for each entry you need, and include the name / corresponding value as needed. Descriptions are optional, but can be used for your own organizational needs. 
-
-**Note:** These properties are just an example. You may need different/more/fewer properties all depending on the SMTP server you’re using.
-
-==============================	==============================
-			Name 							Value
-==============================	==============================
-mail.smtp.auth					true
-mail.smtp.password				[Default User password*]
-mail.smtp.port					[Port number to route through]
-==============================	==============================
-
-**\*WARNING**: Entering a password here will *not* conceal it on-screen. It’s recommended to use an *app password* (for smtp.gmail.com users) or utilize a dedicated/non-personal user account with SMTP server auths so that you do not risk compromising your password.
-
-If your installation’s mail host uses SSL (like smtp.gmail.com) you’ll need these name/value pair properties in place:
-
-======================================	==============================
-				Name 								Value
-======================================	==============================
-mail.smtp.socketFactory.port			465
-mail.smtp.port							465
-mail.smtp.socketFactory.fallback		false
-mail.smtp.socketFactory.class			javax.net.ssl.SSLSocketFactory
-======================================	==============================
-
-The mail session can also be set from command line. To use this method, you will need to delete your notifyMailSession and create a new one. See the below example:
-
-- Delete: ``./asadmin delete-javamail-resource mail/notifyMailSession``
-- Create (remove brackets and replace the variables inside): ``./asadmin create-javamail-resource --mailhost [smtp.gmail.com] --mailuser [test\@test\.com] --fromaddress [test\@test\.com] --property mail.smtp.auth=[true]:mail.smtp.password=[password]:mail.smtp.port=[465]:mail.smtp.socketFactory.port=[465]:mail.smtp.socketFactory.fallback=[false]:mail.smtp.socketFactory.class=[javax.net.ssl.SSLSocketFactory] mail/notifyMailSession``
-
-Be sure you save the changes made here and then restart your Payara server to test it out.
-
 UnknownHostException While Deploying
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

docker-compose-dev.yml

@@ -17,6 +17,8 @@ services:
       SKIP_DEPLOY: "${SKIP_DEPLOY}"
       DATAVERSE_JSF_REFRESH_PERIOD: "1"
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
+      DATAVERSE_MAIL_SYSTEM_EMAIL: "dataverse@localhost"
+      DATAVERSE_MAIL_MTA_HOST: "smtp"
       DATAVERSE_AUTH_OIDC_ENABLED: "1"
       DATAVERSE_AUTH_OIDC_CLIENT_ID: test
       DATAVERSE_AUTH_OIDC_CLIENT_SECRET: 94XHrfNRwXsjqTqApRrwWmhDLDHpIYV8

docker/compose/demo/compose.yml

@@ -14,6 +14,8 @@ services:
       DATAVERSE_DB_PASSWORD: secret
       DATAVERSE_DB_USER: dataverse
       DATAVERSE_FEATURE_API_BEARER_AUTH: "1"
+      DATAVERSE_MAIL_SYSTEM_EMAIL: "Demo Dataverse <dataverse@example.org>"
+      DATAVERSE_MAIL_MTA_HOST: "smtp"
       JVM_ARGS: -Ddataverse.files.storage-driver-id=file1
         -Ddataverse.files.file1.type=file
         -Ddataverse.files.file1.label=Filesystem

modules/container-configbaker/scripts/bootstrap/dev/init.sh

@@ -9,9 +9,6 @@ export DATAVERSE_URL
 echo "Running base setup-all.sh (INSECURE MODE)..."
 "${BOOTSTRAP_DIR}"/base/setup-all.sh --insecure -p=admin1 | tee /tmp/setup-all.sh.out
 
-echo "Setting system mail address..."
-curl -X PUT -d "dataverse@localhost" "${DATAVERSE_URL}/api/admin/settings/:SystemEmail"
-
 echo "Setting DOI provider to \"FAKE\"..."
 curl "${DATAVERSE_URL}/api/admin/settings/:DoiProvider" -X PUT -d FAKE
 

modules/dataverse-parent/pom.xml

@@ -168,11 +168,11 @@
         <gdcc.xoai.version>5.2.0</gdcc.xoai.version>
 
         <!-- Testing dependencies -->
-        <testcontainers.version>1.19.0</testcontainers.version>
-        <smallrye-mpconfig.version>2.10.1</smallrye-mpconfig.version>
-        <junit.jupiter.version>5.10.0</junit.jupiter.version>
-        <mockito.version>5.4.0</mockito.version>
-        <maven-jacoco-plugin.version>0.8.10</maven-jacoco-plugin.version>
+        <testcontainers.version>1.19.7</testcontainers.version>
+        <smallrye-mpconfig.version>3.7.1</smallrye-mpconfig.version>
+        <junit.jupiter.version>5.10.2</junit.jupiter.version>
+        <mockito.version>5.11.0</mockito.version>
+        <maven-jacoco-plugin.version>0.8.11</maven-jacoco-plugin.version>
 
         <checkstyle.version>9.3</checkstyle.version>
 
@@ -182,8 +182,8 @@
         <maven-war-plugin.version>3.3.2</maven-war-plugin.version>
         <maven-dependency-plugin.version>3.5.0</maven-dependency-plugin.version>
         <maven-install-plugin.version>3.1.1</maven-install-plugin.version>
-        <maven-surefire-plugin.version>3.1.0</maven-surefire-plugin.version>
-        <maven-failsafe-plugin.version>3.1.0</maven-failsafe-plugin.version>
+        <maven-surefire-plugin.version>3.2.5</maven-surefire-plugin.version>
+        <maven-failsafe-plugin.version>3.2.5</maven-failsafe-plugin.version>
         <maven-assembly-plugin.version>3.6.0</maven-assembly-plugin.version>
         <maven-resources-plugin.version>3.3.1</maven-resources-plugin.version>
         <maven-release-plugin.version>3.0.0-M7</maven-release-plugin.version>

pom.xml

@@ -51,6 +51,17 @@
                 <artifactId>abdera-i18n</artifactId>
                 <version>1.1.3</version>
             </dependency>
+            <dependency>
+                <groupId>org.apache.abdera</groupId>
+                <artifactId>abdera-parser</artifactId>
+                <version>1.1.3</version>
+                <exclusions>
+                    <exclusion>
+                        <groupId>org.apache.geronimo.specs</groupId>
+                        <artifactId>geronimo-javamail_1.4_spec</artifactId>
+                    </exclusion>
+                </exclusions>
+            </dependency>
         </dependencies>
     </dependencyManagement>
     <!-- Declare any DIRECT dependencies here.
@@ -703,6 +714,7 @@
                     <include>**/firstNames/*.*</include>
                     <include>**/*.xsl</include>
                     <include>**/services/*</include>
+                    <include>**/*.map</include>
                 </includes>
             </resource>
             <resource>

scripts/installer/as-setup.sh

@@ -147,12 +147,10 @@ function final_setup(){
 	# delete any existing mail/notifyMailSession; configure port, if provided:
 
 	./asadmin delete-javamail-resource mail/notifyMailSession
-
-	if [ $SMTP_SERVER_PORT"x" != "x" ]
-	then
-            ./asadmin $ASADMIN_OPTS create-javamail-resource --mailhost "$SMTP_SERVER" --mailuser "dataversenotify" --fromaddress "do-not-reply@${HOST_ADDRESS}" --property mail.smtp.port="${SMTP_SERVER_PORT}" mail/notifyMailSession
-	else
-	    ./asadmin $ASADMIN_OPTS create-javamail-resource --mailhost "$SMTP_SERVER" --mailuser "dataversenotify" --fromaddress "do-not-reply@${HOST_ADDRESS}" mail/notifyMailSession
+	./asadmin $ASADMIN_OPTS create-system-properties "dataverse.mail.system-email='${ADMIN_EMAIL}'"
+  ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.mail.mta.host='${SMTP_SERVER}'"
+	if [ "x${SMTP_SERVER_PORT}" != "x" ]; then
+    ./asadmin $ASADMIN_OPTS create-system-properties "dataverse.mail.mta.port='${SMTP_SERVER_PORT}'"
 	fi
 
 }
@@ -280,6 +278,12 @@ if [ ! -d "$DOMAIN_DIR" ]
     exit 2
 fi
 
+if [ -z "$ADMIN_EMAIL" ]
+ then
+  echo "You must specify the system admin email address (ADMIN_EMAIL)."
+  exit 1
+fi
+
 echo "Setting up your app. server (Payara) to support Dataverse"
 echo "Payara directory: "$GLASSFISH_ROOT
 echo "Domain directory:    "$DOMAIN_DIR

scripts/installer/install.py

@@ -568,14 +568,6 @@
 except:
    sys.exit("Failure to execute setup-all.sh! aborting.")
 
-# 7b. configure admin email in the application settings
-print("configuring system email address...")
-returnCode = subprocess.call(["curl", "-X", "PUT", "-d", adminEmail, apiUrl+"/admin/settings/:SystemEmail"])
-if returnCode != 0:
-   print("\nWARNING: failed to configure the admin email in the Dataverse settings!")
-else:
-   print("\ndone.")
-
 # 8c. configure remote Solr location, if specified
 if solrLocation != "LOCAL":
    print("configuring remote Solr location... ("+solrLocation+")")

scripts/installer/installAppServer.py

@@ -6,8 +6,9 @@ def runAsadminScript(config):
    # commands to set up all the app. server (payara6) components for the application.
    # All the parameters must be passed to that script as environmental                                          
    # variables:
-   os.environ['GLASSFISH_DOMAIN'] = "domain1";
-   os.environ['ASADMIN_OPTS'] = "";
+   os.environ['GLASSFISH_DOMAIN'] = "domain1"
+   os.environ['ASADMIN_OPTS'] = ""
+   os.environ['ADMIN_EMAIL'] = config.get('system','ADMIN_EMAIL')
 
    os.environ['HOST_ADDRESS'] = config.get('glassfish','HOST_DNS_ADDRESS')
    os.environ['GLASSFISH_ROOT'] = config.get('glassfish','GLASSFISH_DIRECTORY')

src/main/docker/scripts/init_2_configure.sh

@@ -31,10 +31,6 @@ echo "# Dataverse postboot configuration for Payara" > "${DV_POSTBOOT}"
 #       EE 8 code annotations or at least glassfish-resources.xml
 # NOTE: postboot commands is not multi-line capable, thus spaghetti needed.
 
-# JavaMail
-echo "INFO: Defining JavaMail."
-echo "create-javamail-resource --mailhost=${DATAVERSE_MAIL_HOST:-smtp} --mailuser=${DATAVERSE_MAIL_USER:-dataversenotify} --fromaddress=${DATAVERSE_MAIL_FROM:-dataverse@localhost} mail/notifyMailSession" >> "${DV_POSTBOOT}"
-
 # 3. Domain based configuration options
 # Set Dataverse environment variables
 echo "INFO: Defining system properties for Dataverse configuration options."