Big sweep: migrate codebase to Jakarta EE 10 and use Payara 6 now (third iteration) #9685

poikilotherm · 2023-06-28T22:06:34Z

~~THIS IS READY BUT SHOULD NOT BE MARKED NON-DRAFT UNTIL 5.14 HAS BEEN RELEASED~~

What this PR does / why we need it:

Payara 5 is going EOL for community usage (= without a payed support contract). Payara 6 is the successor, but allows to use Jakarta EE 10 only. Issue #8305 is the deliverable, where this PR is a part of.

At tech hour we agreed that we want to do a clean cut after the 5.14 release and migrate develop to Jakarta EE 10 and hack on it further. This PR prepares this step.

This PR is a successor to PR #9116

Which issue(s) this PR closes:

Special notes for your reviewer:

Note that as 5.14 is not yet released, this PR will need an update from develop once done and will need to add some more migrations of the outstanding code parts that are due for 5.14. (These are minor, so we started the branch)

Open Tasks this PR did not address (and maybe should be fed back to #8305):

~~Should we preserve backward compatibility of the SWORD API by inserting attachment; for the user if it isn't supplied?~~ DONE, see Big sweep: migrate codebase to Jakarta EE 10 and use Payara 6 now (third iteration) #9685 (comment)
~~The Perl installer has not been updated. Honestly, @pdurbin and I are ok with deleting it (and references to it, of course, if any).~~ DONE, deleted. docker-aio and docker-dcm too.
~~The release note snippet needs work. Upgrades have not been tested.~~ DONE. Release note snippet added. There's a dedicated issue for upgrades: determine upgrade path from Payara 5 to 6 #9340

Suggestions on how to test this:

API tests at https://jenkins.dataverse.org/job/IQSS-Dataverse-Payara6
Will need a great review to make sure we touch every corner of the UI, as Jakarta Faces is notably more strict

Does this PR introduce a user interface change? If mockups are available, please link/include them here:

(No)

Is there a release notes update needed for this change?:

Yes, a stub is included, but needs work. Maybe can be done as part of another issue for Dataverse 6 release notes.

Additional documentation:

None but references in the guides to Payara 5 have been switched to Payara 6.

1. The Jersey dependencies in use did not use the versions imported via the Payara BOM (which itself imports a Jersey BOM). 2. The coordinates for the server package changed, re-aligned.

…nly #8305 This avoids potential trouble were it's unclear from where the higher exceptions come from, javax or jakarta namespace.

https://stackoverflow.com/questions/52737981/rebuilding-primefaces-jar-file-missing-pushservlet

Also add provided dependency of our persistence provider to POM to allow class resolution in IDEs.

Many projects have moved to Eclipse foundation, have been replaced or renamed with the move to Jakarta EE 10. With the import of the new EE 10 Payara BOM, we need to follow their changes (or provide the deps on our own).

As the SPI module used Java EE, we need to release a v2, as the move to Jakarta EE is a breaking change.

Removing some rogue usages of OcpSoft PrettyFaces internal utils for String care and replacing with either Apache Commons Lang3 or native JVM code.

…8305 Any users registered before 2021 release to oss.sonatype.org. As gdcc was installed after that, we only had s01.oss.sonatype.org. Also enabling both snapshot repos for Rewrite and SWORD2 See also https://central.sonatype.org/news/20210223_new-users-on-s01

…pport #8305

…8305 This is due to the fact that Payara 6 Servlets are more strict in their parsing of HTTP headers.

In EditDDI I was seeing a NullPointerException on line 136 (createNewDraftVersion). Switch from `@Context HttpServletRequest` to our ordinary way of API auth fixed it. In DataversesIT I was seeing an OptimisticLockException so I added a sleep.

…a 6 #8305

Involves Jakarta Faces 4.0 and Servlet 6.0

The backing bean actions were attached to normal "<a>" elements, which means they were executed on page load and not, as expected, via JavaScript onClick. Former versions of JSF were less strict, now we need to use the standard compliant "<h:commandLink>"

The Perl installer was removed in b829ca7

kcondon · 2023-08-08T13:57:19Z

Issues found:

Repeated errors thrown in logs when doing nothing and many tasks, publish, nav to dataset page from homepage link, create dataverse, dataset, etc. [KC] Bob saw this in his p5 logs, checking java ver. Leonid will check on his p6 env. [Update] seems isolated to loading a missing/incorrect dataverse so opened a separate issue Error Handling: 404 page thrown as a result of trying to load missing/incorrect Dataverse results in large stack trace but otherwise functions. #9770 , not blocking this one.
idle_log_err.txt
faces_context_large_stack_trace.txt
faces_context_stack_trace.txt

[X] 2. Click dataset page, file kebab, metadata, stack trace but ui ok [KC] Phil is looking into this [Update] fixed, no log error.
file_kebab_metadata_err.txt

Two proposed fixes:

b7ec905: switch to DatasetPage.editFileMetadata()
fedb103: switch to bean.editFileMetadata() and add no-op for Page.editFileMetadata()

[X] 3. Clicking on filename from tree view throws 500 error in ui, saw same context validation errs in log, same error just clicking on file in dataset file table. [KC] Fixed by Oliver.

[X] 4. Possibly related to 2, tabular tags silently can't be added, throws server log err when added from edit metadata popup from files page: [KC] This is fixed now Jim's prior fix was merged/refresh from develop
file_page_tab_tag_select_err.txt

Possibly related to 2, embargo fails with file already published even when dataset hasn't yet been published. Two failure modes from dataset page kebab: a. if file not selected but use file level kebab to embargo, get only error in popup, b. if select file checkbox then choose embargo from file level kebab, get normal embargo popup but with error and no save and server log error.

[embargo_fail.txt](https://github.com/IQSS/dataverse/files/12305211/embargo_fail.txt)

Note: Embargo works from file page
[KC] I've opened a separate issue for this to provide clarity and more room to work: #9773

The backing bean actions were attached to normal "<a>" elements, which means they were executed on page load and not, as expected, via JavaScript onClick. Former versions of JSF were less strict, now we need to use the standard compliant "<h:commandLink>"

poikilotherm · 2023-08-08T21:06:51Z

Issues found:
[...]
3. Clicking on filename from tree view throws 500 error in ui, saw same context validation errs in log, same error just clicking on file in dataset file table.

Should be fixed with 3716c5c.

I scanned all XHTML code for other examples of some <a> with an jsf:action or jsf:actionHandler and could not find other appearances. Looks like dataset and file page citation widget were the only places.

The widget for the dataset citation on the file page was forgotten to be converted to <h:commandLink> as well to avoid action execution on page load.

switch dev-env docs to Docker

landreev · 2023-08-09T22:32:54Z

Regarding item 1. above - those long "illegal State" and "validation failed" exception stack traces are very easy to reproduce: just try accessing a dataverse that does not exist in the database - http://localhost:8080/dataverse/blahblahblah
I'm seeing it in my own payara6 build environment too.

poikilotherm · 2023-08-10T11:40:30Z

Regarding item 1. above - those long "illegal State" and "validation failed" exception stack traces are very easy to reproduce: just try accessing a dataverse that does not exist in the database.

I can verify this procedure and see error logs as well.

As I wrote on internal Slack, this issue is triggered by how the rewrite filter works. From following the execution step by step via debugger, I can see that Jakarta Faces does the abort and the rewrite filter tries to forward us. The first log message is written by the filter detecting we want to abort.

I see an old discussion about this in ocpsoft/rewrite#151. We should create a simpler reproducer and create an issue over there.

For now, the bug does not seem to be fatal, just annoying - the 404 page still works. If I would have a say in this, I'd suggest creating a follow up issue to track this but not let it block merging this PR.

I'm attaching the full output here as well: illegalstate-rewrite-404.txt (from accessing http://localhost:8080/dataverse/test)

scolapasta · 2023-08-10T20:18:53Z

My vote for the kebab metadata issue above would be the 2nd fix: fedb103
as I don't love referencing the DatasetPage on a multi-use bean. I'm also not too particularly concerned, as eventually all this will be going away.

As of Payara 6/JSF 4/Jakarta EE 10, this syntax doesn't work: action="#{bean[editFileAction]()}" (Or it logs errors in server.log at the very least.) So we switch to this syntax: action=#{bean.editFileMetadata()}" This means we have to make sure the method exists on all the beans, though, or we get errors. (It's unclear why we weren't seeing errors before!) So we add the method to FilePage, which gets passed as bean. We also remove the now unused "editFileAction".

github-actions · 2023-08-10T20:39:22Z

📦 Pushed preview images as

ghcr.io/gdcc/dataverse:8305-payara6-ee10-v3

ghcr.io/gdcc/configbaker:8305-payara6-ee10-v3

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

kcondon · 2023-08-11T16:36:42Z

Merging this after discussing on tech. @scolapasta Re: P6 I'm ok with merge now and resolve remaining issues as top priorities.

poikilotherm and others added 27 commits June 28, 2023 16:56

refactor(spi): migrate Dataverse SPI to Jakarta EE 10

ed75148

refactor: migrate main app codebase to Jakarta EE 10

b51d2fc

refactor(test): migrate application tests to Jakarta EE 10

0d60ef2

build!(deps): switch to Primefaces 11 Jakarta Edition #8305

27d8467

build(api): align Jersey deps with BOMs #8305

a1dc896

1. The Jersey dependencies in use did not use the versions imported via the Payara BOM (which itself imports a Jersey BOM). 2. The coordinates for the server package changed, re-aligned.

refactor(webapp): migrate web.xml to Jakarta Faces #8305

cbbb561

docs: fix Jakarta occurences in code comments #8305

c2c75d8

fix(ingest): migrate JMS interface class string to Jakarta #8305

83b0ff4

refactor(api): make Index API check on most specific exception type o…

b74f5d0

…nly #8305 This avoids potential trouble were it's unclear from where the higher exceptions come from, javax or jakarta namespace.

style(webapp): switch URL path to Jakarta Faces #8305

ddda066

fix(webapp): remove deleted Primefaces PushServlet from web.xml #8305

34a421b

https://stackoverflow.com/questions/52737981/rebuilding-primefaces-jar-file-missing-pushservlet

chore(persistence): migrate persistence.xml to Jakarta EE 10 #8305

6bc0330

Also add provided dependency of our persistence provider to POM to allow class resolution in IDEs.

chore(cdi): migrate beans.xml config to Jakarta EE 10 #8305

5fad7e2

chore(build): update to Payara 6.2023.6 #8305

59d38a6

chore(deps): migrate to Jakarta EE 10 dependencies #8305

56f8ad2

Many projects have moved to Eclipse foundation, have been replaced or renamed with the move to Jakarta EE 10. With the import of the new EE 10 Payara BOM, we need to follow their changes (or provide the deps on our own).

chore!(spi): update SPI module to version 2.0.0 #8305

8c5fb1e

As the SPI module used Java EE, we need to release a v2, as the move to Jakarta EE is a breaking change.

build(deps): update to Jakartafied JSON-LD library version #8305

a4a6d23

fix(deps): remove usages of PrettyFaces String utils

e694643

Removing some rogue usages of OcpSoft PrettyFaces internal utils for String care and replacing with either Apache Commons Lang3 or native JVM code.

build(deps): switch to Prettyfaces 6.0 snapshots for Jakarta EE 10 su…

f6772d2

…pport #8305

build(sword): switch to Jakarta-enabled SWORD2 preview release #8305

610c8fd

fix(sword): as of Payara 6, "attachment; " required to upload files #…

89182c1

…8305 This is due to the fact that Payara 6 Servlets are more strict in their parsing of HTTP headers.

Update installer, DockerAIO, Vagrant, some scripts and docs for Payar…

77e35a6

…a 6 #8305

chore(jsf): migrate Faces config and web.xml to Jakarta EE 10 #8305

d882c41

Involves Jakarta Faces 4.0 and Servlet 6.0

fix(ct): optimize base image for Payara 6 #8305

b6659d4

poikilotherm added Component: Containers Anything related to cloudy Dataverse, shipped in containers. D: Payara 6 Upgrade Issues and PRs about the move to Jakarta EE 10 + Payara 6 Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) labels Jun 28, 2023

Merge branch '8305-rm-perl-docker-aio' into 8305-payara6-ee10-v3 #8305

b8bf86a