9434 app container for dev #9439

poikilotherm · 2023-03-13T16:06:17Z

What this PR does / why we need it:

This pull request will add capabilities to create a Dataverse application image and run it along all necessary dependencies.

Which issue(s) this PR closes:

Closes Build locally a Dataverse app image for dev #9434

Special notes for your reviewer:
There are some TODOs:

Get Dataverse communicating with maildev properly
~~(Optional) Maybe add some more resource limitations. Maybe tweak Dataverse container to fit into less than 2 GiB of RAM~~
Add docs (can be copied from 9415 containers #9424 and extend from there). DONE.

Suggestions on how to test this:
Clone/switch. Ensure to have Docker running. Execute mvn -Pct clean package docker:run, then bootstrap when startup is done with ./scripts/dev/docker-final-setup.sh and go to localhost:8080.

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope.

Is there a release notes update needed for this change?:
Yes, included.

Additional documentation:
None yet.

…operties

…ir env var

…hecked

pdurbin · 2023-03-14T01:35:54Z

@poikilotherm this is working great!

Thanks for the heads up about email. I tried using the "Support" link in the header but when I clicked "Send Message" I got this error:

dev_dataverse> Caused by: java.lang.NullPointerException
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.setContactDelegation(MailServiceBean.java:233)

Honestly, I can live without email working but it would be nice to see if we can fix this.

I can also live without the TODO above about limiting resources (for now). Future PR, I'd say. I marked it as optional.

I added docs and a release note. I tried to indicate that this is dev only.

Hopefully the last thing is the SMTP thing. Or, again, I'm fine with fixing later.

There is a TON of value in this PR. Thanks!!

pdurbin · 2023-03-14T11:41:27Z

dev_dataverse> Caused by: java.lang.NullPointerException
dev_dataverse> at edu.harvard.iq.dataverse.MailServiceBean.setContactDelegation(MailServiceBean.java:233)

Here's a stacktrace of the MailDev/SMTP problem: stacktrace.txt

Apparently, fromAddress here is null:

dataverse/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java

Line 233 in a7b0457

String personal = fromAddress.getPersonal() != null

In the domain.xml, the from address is dataverse@localhost:

payara@dataverse:~$ grep mail-resource /opt/payara/appserver/glassfish/domains/domain1/config/domain.xml
<mail-resource auth="false" host="postfix" from="dataverse@localhost" user="dataversenotify" jndi-name="mail/notifyMailSession"></mail-resource>

The from address is not defined as a database setting:

payara@dataverse:~$ curl -s http://localhost:8080/api/admin/settings | jq .
{
  "status": "OK",
  "data": {
    ":Authority": "10.5072",
    ":Shoulder": "FK2/",
    "BuiltinUsers.KEY": "burrito",
    ":DoiProvider": "FAKE",
    ":BlockedApiPolicy": "localhost-only",
    ":Protocol": "doi",
    ":AllowSignUp": "yes",
    ":SignUpUrl": "/dataverseuser.xhtml?editMode=CREATE",
    ":UploadMethods": "native/http"
  }
}

Setting it might fix it.

If I set :SystemEmail like this...

$ curl -X PUT -d 'dataverse@localhost' http://localhost:8080/api/admin/settings/:SystemEmail
{"status":"OK","data":{":SystemEmail":"dataverse@localhost"}}

I get a new error:

dev_dataverse> [#|2023-03-14T11:51:01.758+0000|INFO|Payara 5.2022.4||_ThreadID=87;_ThreadName=http-thread-pool::http-listener-1(3);_TimeMillis=1678794661758;_LevelValue=800;|
dev_dataverse>   com.sun.mail.util.MailConnectException: Couldn't connect to host, port: postfix, 25; timeout -1;
dev_dataverse>   nested exception is:
dev_dataverse> 	java.net.ConnectException: Connection refused (Connection refused)
dev_dataverse> 	at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2209)
dev_dataverse> 	at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:722)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:342)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:222)
dev_dataverse> 	at javax.mail.Service.connect(Service.java:171)
dev_dataverse> 	at javax.mail.Transport.send0(Transport.java:230)
dev_dataverse> 	at javax.mail.Transport.send(Transport.java:100)
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.sendMail(MailServiceBean.java:217)
dev_dataverse> 	at edu.harvard.iq.dataverse.MailServiceBean.sendMail(MailServiceBean.java:181)

From my Mac, the SMTP server is running on port 25...

pdurbin@air dataverse % nc localhost 25
220 postfix ESMTP

... but the dataverse-1 (app container) can't reach it:

payara@dataverse:~$ nc postfix 25
payara@dataverse:~$

Why?

docker-compose-dev.yml

GPortas · 2023-03-14T13:39:45Z

@poikilotherm

I have done the following steps so far:

Built the image through mvn -Pct clean package
Executed the containers through docker-compose -f docker-compose-dev.yml up
Executed ./scripts/dev/docker-final-setup.sh when startup is done

Received this error within the docker-compose logs while applying the third step:

dev_dataverse_1         | [#|2023-03-14T13:25:45.002+0000|WARNING|Payara 5.2022.4|javax.enterprise.ejb.container|_ThreadID=81;_ThreadName=http-thread-pool::http-listener-1(1);_TimeMillis=1678800345002;_LevelValue=900;|
dev_dataverse_1         |   javax.ejb.EJBException: getSingleResult() did not retrieve any entities.
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.processSystemException(EJBContainerTransactionManager.java:723)
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:652)
dev_dataverse_1         | 	at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:482)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4601)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2134)
dev_dataverse_1         | 	at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2104)

. . .

Dataverse runs on 8080, but when trying to Log In with dataverseAdmin / admin credentials, the authentication fails. Am I missing something?

poikilotherm · 2023-03-14T14:10:45Z

@GPortas please note that the login is "dataverseAdmin:admin1". That "1" is easy to forget.

Also, the "getSingleResult()" thing is very annoying but harmless. For some reason no one ever fixed this also it would be a low hanging fruit. Same goes for all the SQL exceptions which are triggered by us generating the DDL everytime we start the application.

All containers have static names now, which might be a problem down the road. But at least consistent now.

docker-compose-dev.yml

johannes-darms · 2023-03-16T09:18:44Z

The approach is also missing R and Rserve. They are listed as a prerequisite within the documentation but it looks like they are not really needed. If R/RServe is optional the documentation must be updated, otherwise another container wrapping RServe is needed.

https://guides.dataverse.org/en/latest/installation/prerequisites.html?highlight=rserve

poikilotherm · 2023-03-16T09:33:25Z

Thanks @johannes-darms for the extensive review of the PR! I added comments to your comments.

The approach is also missing R and Rserve. They are listed as a prerequisite within the documentation but it looks like they are not really needed. If R/RServe is optional the documentation must be updated, otherwise another container wrapping RServe is needed.

https://guides.dataverse.org/en/latest/installation/prerequisites.html?highlight=rserve

Two things here: 1) Rserve is not a necessary component for developing Dataverse, which is what this is about. 2) Technically it even is not a requirement for an installation if you don't do any ingest. But yes, that should be stated in the docs. Would you be able to create an issue and/or a pull request?

pdurbin · 2023-03-16T19:22:59Z

not a necessary component for developing Dataverse, which is what this is about

Good point. @poikilotherm how do you feel about a new title for this PR?

"app container" -> "app container for dev"

@johannes-darms thanks for all the feedback! I also left comments on your comments. Much appreciated!!

GPortas · 2023-03-17T12:43:11Z

@poikilotherm

Although the asadmin commands are enough to manage the Payara server, sometimes it is very useful to access the Payara administration console for better visualization.

By default, Payara exposes the administration console on port 4848. It would be interesting to map this port to outside the container within the docker-compose file.

I've tested this with this branch, by adding - "4848:4848" to the dev_dataverse mapped ports.

The problem is that the admin console now expects user and password credentials to login (As far as I know, these credentials are not documented). With the normal installation (without containers), we don't need credentials for accessing the Payara admin console. Is that login requirement a configuration that comes from the base image? It would be great to make admin credentials configurable through container environment variables.

poikilotherm · 2023-03-17T12:48:44Z

@GPortas yes, this comes from the base image. We could add a section to the base image documentation, similar to upstream Payara as this works exactly the same in our custom base image.

In fact, the credentials are available via the (currently undocumented) env vars ADMIN_USER and ADMIN_PASSWORD, but changing these env vars doesn't update the running domain.

@GPortas probably this is beyond scope for this PR. Would you add a note to https://docs.google.com/document/d/15-sqdKzpCgQBtaPaAGYMaqcxsQXmt-2CwAVNPmmAPRY so we don't forget?

poikilotherm · 2023-03-17T13:47:13Z

@pdurbin @johannes-darms I added a few pieces of information about a future sunset of docker-aio in 7bcadab. Let me know if that helps. Feel free to hack on it.

pdurbin · 2023-03-17T18:10:24Z

a future sunset of docker-aio

Moving docker-dcm to this repo sbgrid/data-capture-module#39

kcondon · 2023-03-20T14:17:45Z

Issues so far:

Running into build failure due to test failures. Am I doing something wrong?
app_contain_bld_fail.txt
[Kevin] I think it was because I didn't do switch -c branch, gets further now, see issue 2
Mvn script doesn't complete, pauses, shows errors. Ran set up script and localhost:8080 shows only Payara.
mvn_err_pause.txt
final_setup_run.txt

poikilotherm · 2023-03-20T15:24:45Z

Hi @kcondon, thx for trying this. The log says that the database connection could not be established. It looks like you already had the local volumes from before (most likely #9414 / #9417), so there might be a different database already in place. Could you delete the whole ./docker-dev-volumes folder and try again? Thx!

Without setting the "env var" for docker compose within the Maven POM, the value will not be set, leading to a broken DB connection. The DMP does not read the values from .env.

poikilotherm · 2023-03-20T19:51:37Z

Woo-hoo thanks for merging @kcondon ! Much appreciated, as always!!

poikilotherm added 8 commits March 13, 2023 15:11

fix(ct): make directory config source override microprofile-config.pr…

76f1a74

…operties

chore(deps): update docker maven plugin to 0.42.0

519b7d0

feat(ct): make temp file storage location in containers use storage d…

65f2478

…ir env var

feat(ct): add configure script hack for Payara to get going

7374bb7

feat(ct): add application Dockerfile and assembly config

21e8d72

feat(ct): add cap to build images for the application from Maven

5f49e68

feat(ct): extend compose file to include and start application

dde54df

feat(ct): add simple bootstrap script inspired by @pdurbin but shellc…

b052423

…hecked

poikilotherm requested review from pdurbin and GPortas March 13, 2023 16:06

poikilotherm self-assigned this Mar 13, 2023

This was referenced Mar 13, 2023

9415 containers #9424

Closed

Analyze Dataverse application containerization for development #9415

Closed

Build locally a Dataverse app image for dev #9434

Closed

pdurbin added this to Ready for Review ⏩ in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) via automation Mar 13, 2023

pdurbin unassigned poikilotherm Mar 13, 2023

add docs, rename script #9434

a7b0457

pdurbin moved this from Ready for Review ⏩ to Review 🔎 in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Mar 14, 2023

pdurbin self-assigned this Mar 14, 2023

GPortas self-assigned this Mar 14, 2023

GPortas reviewed Mar 14, 2023

View reviewed changes

docker-compose-dev.yml Outdated Show resolved Hide resolved

fix(ct): add missing DV container name in docker compose file

47d6ed4

All containers have static names now, which might be a problem down the road. But at least consistent now.

johannes-darms reviewed Mar 16, 2023

View reviewed changes

docker-compose-dev.yml Show resolved Hide resolved

poikilotherm changed the title ~~9434 app container~~ 9434 app container for dev Mar 16, 2023

doc(ct): add note about future sunset of docker-aio

7bcadab

pdurbin mentioned this pull request Mar 17, 2023

Moving docker-dcm to this repo sbgrid/data-capture-module#39

Open

feat(ct): make app image name overrideable from Maven build

bf51294

kcondon moved this from Ready for QA ⏩ to QA ✅ in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Mar 20, 2023

kcondon self-assigned this Mar 20, 2023

kcondon moved this from QA ✅ to Ready for QA ⏩ in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Mar 20, 2023

kcondon removed their assignment Mar 20, 2023

kcondon moved this from Ready for QA ⏩ to QA ✅ in IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) Mar 20, 2023

kcondon self-assigned this Mar 20, 2023

fix(ct): add missing db user var to POM

b99073c

Without setting the "env var" for docker compose within the Maven POM, the value will not be set, leading to a broken DB connection. The DMP does not read the values from .env.

kcondon merged commit a9a8311 into develop Mar 20, 2023

IQSS/dataverse (TO BE RETIRED / DELETED in favor of project 34) automation moved this from QA ✅ to Done 🚀 Mar 20, 2023

kcondon deleted the 9434-app-container branch March 20, 2023 19:17

pdurbin mentioned this pull request Mar 24, 2023

Support for deleting files using native API #9383

Merged

GPortas mentioned this pull request Mar 27, 2023

Make Payara Admin Console accessible in the containerized setup #9476

Closed

GPortas mentioned this pull request Apr 12, 2023

Solr custom Docker image for dev #9516

Closed

pdurbin added this to the 5.14 milestone May 10, 2023

helkv mentioned this pull request Aug 16, 2023

Edmond-Dataverse in Docker MPDL/dataverse#71

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

9434 app container for dev #9439

9434 app container for dev #9439

poikilotherm commented Mar 13, 2023 •

edited by pdurbin

pdurbin commented Mar 14, 2023

pdurbin commented Mar 14, 2023 •

edited

GPortas commented Mar 14, 2023 •

edited

poikilotherm commented Mar 14, 2023

johannes-darms commented Mar 16, 2023

poikilotherm commented Mar 16, 2023

pdurbin commented Mar 16, 2023

GPortas commented Mar 17, 2023 •

edited

poikilotherm commented Mar 17, 2023 •

edited

poikilotherm commented Mar 17, 2023

pdurbin commented Mar 17, 2023

kcondon commented Mar 20, 2023 •

edited

poikilotherm commented Mar 20, 2023 •

edited

poikilotherm commented Mar 20, 2023

9434 app container for dev #9439

9434 app container for dev #9439

Conversation

poikilotherm commented Mar 13, 2023 • edited by pdurbin

pdurbin commented Mar 14, 2023

pdurbin commented Mar 14, 2023 • edited

GPortas commented Mar 14, 2023 • edited

poikilotherm commented Mar 14, 2023

johannes-darms commented Mar 16, 2023

poikilotherm commented Mar 16, 2023

pdurbin commented Mar 16, 2023

GPortas commented Mar 17, 2023 • edited

poikilotherm commented Mar 17, 2023 • edited

poikilotherm commented Mar 17, 2023

pdurbin commented Mar 17, 2023

kcondon commented Mar 20, 2023 • edited

poikilotherm commented Mar 20, 2023 • edited

poikilotherm commented Mar 20, 2023

poikilotherm commented Mar 13, 2023 •

edited by pdurbin

pdurbin commented Mar 14, 2023 •

edited

GPortas commented Mar 14, 2023 •

edited

GPortas commented Mar 17, 2023 •

edited

poikilotherm commented Mar 17, 2023 •

edited

kcondon commented Mar 20, 2023 •

edited

poikilotherm commented Mar 20, 2023 •

edited