as a developer, I'd like the docker-aio plumbing to work.

[pameyer]

At some point, conf/docker-aio/prep_it.bash broke and no longer results in a running dataverse instance for running ITs on. Starting glassfish in the container after prep_it.bash does result in a working configuration.

This issue is more for future reference, in the event anyone else runs into it. Shouldn't be a big fix, not likely to dig into it in the immediate future.

[poikilotherm]

Don't throw stones on me, but maybe, just maybe, instead of fixing this it might be an idea to switch to Arquillian for IT tests, once my Docker work is done. Adding a ref to this here: #5068

[pdurbin]

@poikilotherm are you using Arquillian in PeerPub? I don't see it in the pom at https://github.com/peerpub/peerpub . Either way switching from REST Assured to Arquillian is out of scope for this issue @pameyer opened but it certainly doesn't hurt to mention the idea! 😄

[pameyer]

@poikilotherm If docker-aio becomes obsolete because there's a better way, then this won't be something that needs fixing :)

[poikilotherm]

@poikilotherm are you using Arquillian in PeerPub? I don't see it in the pom at https://github.com/peerpub/peerpub .

No, the project wasn't as far and not as complex as Dataverse is. You integrate way more stuff than PeerPub was ever thought to be using... 😉

Either way switching from REST Assured to Arquillian is out of scope for this issue @pameyer opened but it certainly doesn't hurt to mention the idea!

Don't get me wrong - you still need test cases e. g. with REST Assured! Arquillian is just about automation and control of 'em.

[pameyer]

From some investigation with the normal installer in docker-aio/docker-dcm, it appears that this may be related to either docker oddness or something looking for an interactive shell in the installation script. docker exec -it dv /opt/dv/install.bash results in no glassfish process inside the container; docker exec -it dv bash followed by /opt/dv/install.bash (inside the container) behave as expected with glassfish still running after installation.

[pdurbin]

I finally got around to trying this (on 5a24bc8 on "develop") and I had a similarly bad time.

The output of ./conf/docker-aio/prep_it.bash looked fine up for a while...

Giving @spruce admin on Birds dataverse
{"status":"OK","data":{"assignee":"@spruce","roleId":1,"_roleAlias":"admin","definitionPointId":2}}

{
  "assignee": "@spruce",
  "_roleAlias": "admin"
}

... but the errors started with publishing datasets? Here:

 <sword:error href="Unable to publish dataset: edu.harvard.iq.dataverse.engine.command.exception.CommandException: This dataset may not be published due to an error when contacting the &lt;a href=http://status.datacite.org target=&quot;_blank&quot;/&gt; DataCite &lt;/a&gt; Service. Please try again." xmlns:sword="http://purl.org/net/sword/terms/"><atom:title xmlns:atom="http://www.w3.org/2005/Atom">ERROR</atom:title><atom:updated xmlns:atom="http://www.w3.org/2005/Atom">2019-01-25T20:21:14Z</atom:updated><atom:generator uri="http://www.swordapp.org/" version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"/><sword:treatment>Processing failed</sword:treatment><atom:summary xmlns:atom="http://www.w3.org/2005/Atom">Unable to publish dataset: edu.harvard.iq.dataverse.engine.command.exception.CommandException: This dataset may not be published due to an error when contacting the &lt;a href=http://status.datacite.org target="_blank"/&gt; DataCite &lt;/a&gt; Service. Please try again.</atom:summary></sword:error>

<?xml version="1.0" encoding="ISO-8859-1"?>
<sword:error xmlns:sword="http://purl.org/net/sword/terms/" href="Unable to publish dataset: edu.harvard.iq.dataverse.engine.command.exception.CommandException: This dataset may not be published due to an error when contacting the &lt;a href=http://status.datacite.org target=&quot;_blank&quot;/&gt; DataCite &lt;/a&gt; Service. Please try again.">
  <atom:title xmlns:atom="http://www.w3.org/2005/Atom">ERROR</atom:title>
  <atom:updated xmlns:atom="http://www.w3.org/2005/Atom">2019-01-25T20:21:14Z</atom:updated>
  <atom:generator xmlns:atom="http://www.w3.org/2005/Atom" uri="http://www.swordapp.org/" version="2.0"/>
  <sword:treatment>Processing failed</sword:treatment>
  <atom:summary xmlns:atom="http://www.w3.org/2005/Atom">Unable to publish dataset: edu.harvard.iq.dataverse.engine.command.exception.CommandException: This dataset may not be published due to an error when contacting the &lt;a href=http://status.datacite.org target="_blank"/&gt; DataCite &lt;/a&gt; Service. Please try again.</atom:summary>
</sword:error>
setupIT worked (0 tries)
Remote server does not listen for requests on [localhost:4848]. Is the server up?
No such local command: create-jvm-options.  Unable to access the server to execute the command remotely.  Verify the server is available. 
Command create-jvm-options failed.
seturl fail; bailing out

By the end Glassfish isn't running. I'm not sure why this would be related to datasets not publishing. Maybe it isn't.

Anyway, I tried starting Glassfish like this:

murphy:dataverse pdurbin$ docker exec -it dv /usr/local/glassfish4/bin/asadmin start-domain
Waiting for domain1 to start ...........................................................
Successfully started the domain : domain1
domain  Location: /opt/glassfish4/glassfish/domains/domain1
Log File: /opt/glassfish4/glassfish/domains/domain1/logs/server.log
Admin Port: 4848
Command start-domain executed successfully.
murphy:dataverse pdurbin$ 

... then I went to http://localhost:8084 but Glassfish wasn't up like I expected. So I tried to list the running applications but Glassfish was down again already:

murphy:dataverse pdurbin$ docker exec -it dv /usr/local/glassfish4/bin/asadmin list-applications
Remote server does not listen for requests on [localhost:4848]. Is the server up?
No such local command: list-applications.  Unable to access the server to execute the command remotely.  Verify the server is available. 
Command list-applications failed.
murphy:dataverse pdurbin$ 

So yeah, this is pretty broken, which is unfortunate because we learn more heavily that we should on @pameyer using all this great docker-aio code he wrote to catch all kinds of bug before pull requests get merged.

[pameyer]

Did a little more investigation; currently leaning towards a change in TTY behavior between centos 7.6 and 7.5.

[pdurbin]

Some more info at #5662 (comment)

[pameyer]

From process of elimination, it appears that the external setupIT -> install -> glassfish-setup.sh is where glassfish ends up not running. glassfish-setup.sh runs to completion though.

At this point, I'm suspecting something related to the change from asadmin restart-domain to asadmin stop-domain ; asadmin start-domain (which if memory serves, was because the restart wasn't fully stopping before starting).

[pameyer]

Current thinking is that the TTY relatedness was the right track, but not due to any centos related changes. #5294 changed an asadmin restart-domain to asadmin stop-domain ; asadmin start-domain. Without diving into appserver-cli.jar, it appears that when a tty disappears shortly after the asadmin command returns, the glassfish process dies. This interpretation is consistent with the previous (desired) behavior of prep_it to have been working by accident.

[pdurbin]

I tried the updated readme in pull request #5699 (the two commands at the top) and once again I'm able to use docker-aio to run the API test suite. Thank you, @pameyer !

I'll note that the test suite failed with testMakeDataCountGetMetric failure reported elsewhere and in #5686 (comment) I explained the scp workaround I implemented on the Jenkins server to make the phoenix server happy.

[pdurbin]

I'll note that the test suite failed with testMakeDataCountGetMetric failure

I just created #5702 to track this. As I've said elsewhere I am unable to run the full API test suite against Glassfish running on my Mac laptop because the tests crash Glassfish. 😞