configbaker container by poikilotherm · Pull Request #9574 · IQSS/dataverse

poikilotherm · 2023-05-04T21:48:25Z

What this PR does / why we need it:

This pull request will add a new container to be built and pushed alongside the application itself, but as a standalone thing to pull from a registry. This image contains lots of tools and scripts necessary to successfully start and configure a Dataverse instance to a) avoid repeating these instructions in different flavors over and over again plus b) encapsulate the Linux scripts in a container so it can be executed on any platform.

Which issue(s) this PR closes:

Special notes for your reviewer:
None yet. This is a WIP.

Suggestions on how to test this:
docker run -it --rm gdcc/configbaker:unstable

Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope.

Is there a release notes update needed for this change?:
Ahem. Dunno. Maybe.

Additional documentation:
None yet.

pdurbin

Just some initial feedback. I ran it with no arguments and got the help ok.

To make use of the setup scripts inside containers, the target URL must be configurable and not hardcoded to localhost:8080. This commit simply creates a variable and uses the old static URL as default if it has not been set by the environment before.

This commit enhances any reference to other files (scripts or data) to be relative to the script currently running instead of the working directory from where the script is called from. This removes the requirement to change into the directory with setup-all.sh first.

pdurbin

Looks fine but I haven't tested it.

pdurbin · 2023-05-09T14:22:31Z

+export DATAVERSE_URL
+SOLR_URL=${SOLR_URL:-"http://localhost:8983"}
+
+SCRIPT_PATH="$(dirname "$0")"


Suggested change

SCRIPT_PATH="$(dirname "$0")"

# scripts/api when called from the root of the source tree

SCRIPT_PATH="$(dirname "$0")"

Copying the default configset of the requested Solr version and adding our schema and config creates a marvelous template. To be reused during execution of a vanilla Solr image.

By building the image at the same time as the application, we can ensure both match. It also keeps compatibility between the compose file to start things and the Maven build.

With adding bootstrap.sh, a Dataverse instance may now be initialized. Using the changed setup scripts, we can reach out to a Dataverse container and set it up. By using a concept of personas, it is easy to add and select other initializations as necessary. This will be extended in the future.

coveralls · 2023-05-09T16:17:04Z

Coverage: 20.138% (-0.03%) from 20.171% when pulling 051d132 on poikilotherm:9573-configbaker into 58dac8d on IQSS:develop.

Now an instance can be bootstrapped automatically.

pdurbin

Just a quick question.

pdurbin · 2023-05-10T00:31:15Z

 echo "deleting all data from Solr"
-curl http://localhost:8983/solr/collection1/update/json?commit=true -H "Content-type: application/json" -X POST -d "{\"delete\": { \"query\":\"*:*\"}}"
-
-SERVER=http://localhost:8080/api


Is it strictly necessary to remove the hard coded 8080 in this and other setup scripts?

I'm awfully sorry me lad, but yes it is.

When you run this script inside a container which is not the application server itself, localhost is that container and not the app server. It's like a different machine. If you run this on the host, this works because of the mapped ports, but this will not work on a Windows host.

Sorry, hardcoding it never was a good idea, same as the paths not being relative to the script location.

BTW Jenkins made it to the end, so I'm pretty sure I didn't break the scripts 😄

pdurbin · 2023-05-10T00:32:55Z

I tried this as of 36a43ac and it worked great!

It's now literally a one liner to spin up a Dataverse dev environment:

mvn -Pct clean package docker:run

Great work, @poikilotherm! 🎉 🚀

This was added long ago in 42692f5 when we were still indexing users! We don't need it anymore. If people have junk in Solr they can clear it out manually before installing Dataverse.

…text

…mpose

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

poikilotherm · 2023-05-30T21:43:17Z

/push-image

Test what happens with the configbaker image...?

github-actions · 2023-05-30T21:46:42Z

📦 Could not push preview image 😞. See log for details.

poikilotherm · 2023-05-30T21:58:03Z

/push-image

Try again to push the images after fixing the registry target in develop.

github-actions · 2023-05-30T22:03:17Z

📦 Pushed preview application image as

ghcr.io/gdcc/dataverse:9573-configbaker

🚢 See on GHCR. Use by referencing with full name as printed above, mind the registry name.

…escription

pdurbin

Looks good!

As I mentioned in standup, setup-all has been updated but I'm not too worried because it is exercised by Jenkins.

feat(ct): add first elements of gdcc/configbaker

a55d6b6

poikilotherm added the Component: Containers Anything related to cloudy Dataverse, shipped in containers. label May 4, 2023

poikilotherm self-assigned this May 4, 2023

feat(ct): print script descriptions in help of configbaker

232b1af

pdurbin reviewed May 5, 2023

View reviewed changes

poikilotherm added 2 commits May 9, 2023 16:06

pdurbin reviewed May 9, 2023

View reviewed changes

poikilotherm added 3 commits May 9, 2023 16:31

feat(ct): add Solr configset to configbaker

f7a7c42

Copying the default configset of the requested Solr version and adding our schema and config creates a marvelous template. To be reused during execution of a vanilla Solr image.

refactor(ct): make configbaker buildable via Maven

b9ea230

By building the image at the same time as the application, we can ensure both match. It also keeps compatibility between the compose file to start things and the Maven build.

feat(ct): add configbaker based bootstrap init to compose

ceddf70

Now an instance can be bootstrapped automatically.

poikilotherm force-pushed the 9573-configbaker branch from 051d132 to ceddf70 Compare May 9, 2023 16:18

Merge branch 'develop' into 9573-configbaker

36a43ac

pdurbin reviewed May 10, 2023

View reviewed changes

pdurbin added the Size: 3 A percentage of a sprint. 2.1 hours. label May 10, 2023

johannes-darms reviewed May 16, 2023

View reviewed changes

Comment thread scripts/api/setup-all.sh Outdated

johannes-darms reviewed May 16, 2023

View reviewed changes

Comment thread modules/container-configbaker/Dockerfile

pdurbin changed the title ~~9573 configbaker container~~ configbaker container May 25, 2023

pdurbin added 2 commits May 25, 2023 14:14

add docs IQSS#9573

920770a

add comments to scripts IQSS#9573

ae87556

pdurbin self-assigned this May 25, 2023

pdurbin and others added 6 commits May 25, 2023 14:35

simplify dev usage, add quickstart IQSS#9573

9e4fcce

stop clearing out Solr on install IQSS#9573

13f92f0

This was added long ago in 42692f5 when we were still indexing users! We don't need it anymore. If people have junk in Solr they can clear it out manually before installing Dataverse.

Merge branch 'develop' into 9573-configbaker

63811cc

chore(build): update Docker Maven Plugin to 0.43.0

590763b

feat(ct): make configbaker bootstrap list available personas in help …

e8ebd8f

…text

ci: add configbaker to shellcheck workflow

4014438

poikilotherm and others added 5 commits May 30, 2023 22:42

docs(ct): extend description tag of configbaker image

16ea1d1

build(ct): skip running a second bootstrap container

d672e32

feat(ct): switch compose file to use configbaker for Solr

969bd2f

fix(ct): make configbaker solr initializer command compatible with co…

36e4e87

…mpose

docs(ct): fix source tag for configbake image

415cfd4

Co-authored-by: Philip Durbin <philipdurbin@gmail.com>

poikilotherm added 2 commits May 30, 2023 23:47

Merge branch 'develop' into 9573-configbaker

8f7c1d8

Merge branch 'develop' into 9573-configbaker

c8198a7

poikilotherm added 7 commits May 31, 2023 00:17

ci(ct): push description of configbaker to Docker Hub, too

044e93a

ci(ct): fix wording to include configbaker, too.

bf0a43f

docs(solr,ct): add info comment to field update script

cfe167a

feat(ct): include Solr field update script in configbaker

32f0977

docs(admin): add anchor to metadata block customization solr update d…

fdc9b39

…escription

docs(ct): extend container guide on configbaker image

b37d501

docs(ct): make renaming of ConfigBaker to Config Baker consistent

47f949d

poikilotherm marked this pull request as ready for review May 31, 2023 14:14

small doc tweaks IQSS#9573

3ab1394

pdurbin approved these changes May 31, 2023

View reviewed changes

pdurbin unassigned pdurbin and poikilotherm May 31, 2023

kcondon self-assigned this Jun 1, 2023

kcondon merged commit 30e9ef5 into IQSS:develop Jun 1, 2023

pdurbin added this to the 5.14 milestone Jun 1, 2023

pdurbin mentioned this pull request Sep 7, 2023

Configurable server-port-number for the containerized Dataverse #9534

Closed

poikilotherm deleted the 9573-configbaker branch October 2, 2024 06:38

cmbz added FY26 Sprint 14 FY26 Sprint 14 (2025-12-31 - 2026-01-14) and removed FY26 Sprint 14 FY26 Sprint 14 (2025-12-31 - 2026-01-14) labels Jan 14, 2026

	SCRIPT_PATH="$(dirname "$0")"
	# scripts/api when called from the root of the source tree
	SCRIPT_PATH="$(dirname "$0")"

Conversation

poikilotherm commented May 4, 2023 • edited by pdurbin Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

pdurbin May 9, 2023

Choose a reason for hiding this comment

Uh oh!

coveralls commented May 9, 2023

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

pdurbin May 10, 2023

Choose a reason for hiding this comment

Uh oh!

poikilotherm May 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

poikilotherm May 10, 2023

Choose a reason for hiding this comment

Uh oh!

pdurbin commented May 10, 2023

Uh oh!

Uh oh!

Uh oh!

poikilotherm commented May 30, 2023

Uh oh!

github-actions Bot commented May 30, 2023

Uh oh!

poikilotherm commented May 30, 2023

Uh oh!

github-actions Bot commented May 30, 2023

Uh oh!

pdurbin left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

poikilotherm commented May 4, 2023 •

edited by pdurbin

Loading

poikilotherm May 10, 2023 •

edited

Loading