Add minimal CouchDB container for WMAgent #1409

todor-ivanov · 2023-08-18T17:08:58Z

fixing dmwm/WMCore#11312

This PR adds all necessary changes for setting up a WMAgent couchdb container.

The default CouchDB version is 3.2.2, but with the help of the wrapper scripts any couchdb version can be built and uploaded to registry.cern.ch
The containers are uploaded at:
- registry: registry.cern.ch
- repository: [cmsweb/wmagent-couchdb](https://registry.cern.ch/harbor/projects/1771/repositories/wmagent-couchdb)

It is ready to use.

Remove anything unnecessary Remove local-upstream-minimal.ini

… broken tag build parameter

… security string checks from manage script.

…ted here

khurtado · 2024-03-07T14:17:12Z

@todor-ivanov Should we add a mechanism to read the couch user/pass from a config file ? Read my comment from:

dmwm/WMCore#11312 (comment)

todor-ivanov · 2024-03-07T18:24:28Z

Hi @khurtado we should indeed. I am workign on that now, and hopefully will provide the code later tonight.
Sorry for not replying ... I was concentrating the Oracle functionalities fro wmagent.

khurtado · 2024-03-07T21:33:36Z

@todor-ivanov Sounds good! I will test on my end once you are done but let me know if you need help with anything.

…CH}_SECRETS_FILE && Start couchDB from run.sh

todor-ivanov · 2024-03-11T13:21:24Z

FYI: dmwm/WMCore#11312 (comment) @khurtado

khurtado · 2024-03-12T00:39:31Z

@todor-ivanov I was testing the options in the script and this one didn't work for me:

(CouchDB-3.2.2) [cmst1@vocms0265:data]$ manage updatecouchapps
ME : 3.2.2
TOP : /data
ROOT : /data/srv
CFGDIR : /data/srv/couchdb/3.2.2/config
LOGDIR : /data/srv/couchdb/3.2.2/logs
STATEDIR : /data/srv/couchdb/3.2.2/state
KEYFILE : /data/srv/couchdb/auth//hmackey.ini

COUCH_ROOT_DIR : /data
COUCH_BASE_DIR : /data/srv/couchdb
COUCH_STATE_DIR : /data/srv/couchdb/3.2.2/state
COUCH_INSTALL_DIR : /data/srv/couchdb/3.2.2/install
COUCH_CONFIG_DIR : /data/srv/couchdb/3.2.2/config

/data/srv/couchdb/3.2.2/install/stagingarea/tmp /data

Pulling couchapps version  from Github...
https://codeload.github.com/dmwm/WMCore/tar.gz/refs/tags/:
2024-03-12 00:38:14 ERROR 400: Bad Request.
Error pulling couchapps version  from Github

todor-ivanov · 2024-03-13T12:47:26Z

hi @khurtado

I was testing the options in the script and this one didn't work for me:
(CouchDB-3.2.2) [cmst1@vocms0265:data]$ manage updatecouchapps

You need to provide the WMCore tag from which the relevant coucapps should be updated:

(CouchDB-3.2.2) [cmst1@vocms0290:data]$ manage updatecouchapps 2.3.0
ME : 3.2.2
TOP : /data
ROOT : /data/srv
CFGDIR : /data/srv/couchdb/3.2.2/config
LOGDIR : /data/srv/couchdb/3.2.2/logs
STATEDIR : /data/srv/couchdb/3.2.2/state
KEYFILE : /data/srv/couchdb/auth//hmackey.ini

COUCH_ROOT_DIR : /data
COUCH_BASE_DIR : /data/srv/couchdb
COUCH_STATE_DIR : /data/srv/couchdb/3.2.2/state
COUCH_INSTALL_DIR : /data/srv/couchdb/3.2.2/install
COUCH_CONFIG_DIR : /data/srv/couchdb/3.2.2/config

/data/srv/couchdb/3.2.2/install/stagingarea/tmp /data

Pulling couchapps version 2.3.0 from Github...
2024-03-13 12:45:01 URL:https://codeload.github.com/dmwm/WMCore/tar.gz/refs/tags/2.3.0 [11592963] -> "2.3.0.tar.gz" [1]

Pulling additional reqmon and t0_reqmon dependencies...

Downloading jquery-ui.min.js...
2024-03-13 12:45:02 URL:https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.18/jquery-ui.min.js [201842/201842] -> "jquery-ui.min.js" [1]

Downloading jquery.min.js...
2024-03-13 12:45:02 URL:http://code.jquery.com/jquery-1.7.2.min.js [94840/94840] -> "jquery-1.7.2.min.js" [1]

Downloading Datatables...
2024-03-13 12:45:02 URL:https://datatables.net/releases/DataTables-1.9.1.zip [2415658/2415658] -> "DataTables-1.9.1.zip" [1]

Downloading YUI...
2024-03-13 12:45:03 URL:https://yui.github.io/yui2/archives/yui_2.9.0.zip [14294111/14294111] -> "yui_2.9.0.zip" [1]
/data/srv/couchdb/3.2.2/install/stagingarea/tmp/yui /data/srv/couchdb/3.2.2/install/stagingarea/tmp /data
/data/srv/couchdb/3.2.2/install/stagingarea/tmp /data
Removing old couchapps...
Installing new couchapps...
/data
Cleaning up!

todor-ivanov · 2024-03-13T12:49:00Z

@khurtado @amaltaro

Please take a look at the final version of this setup.
I think it is now working as expected.

…nly admin section Typo

khurtado

@todor-ivanov Code is looking good to me, I left a few comments. Just one thing overall, this PR is in general not covering non-cmst1 users, right? I vaguely remember CERN IT was going to be involved in this, security wise (regarding including uuids and usernames of all three users Agents use)

I updated the documentation suggestion with the updatecouch output in your previous comment.

docker/pypi/wmagent-couchdb/README.md

docker/pypi/wmagent-couchdb/couchdb-docker-build.sh

docker/pypi/wmagent-couchdb/couchdb-docker-run.sh

khurtado · 2024-03-15T00:04:25Z

@todor-ivanov Just a reminder of this review for when you get a change to look at it

Co-authored-by: Kenyi Hurtado <khurtado@nd.edu>

todor-ivanov · 2024-03-18T10:21:27Z

Hi @khurtado I have merged all your suggeted changes to the current PR. I also addressed your comments and also added the needed push and pull parameters to the scripts in order to run upload and run this image to/from the cern registry. Please take another look.

Forgotten extra repository variable

khurtado · 2024-03-18T19:35:40Z

@todor-ivanov
I think the unused var COUCH_SECRETS_FILE was not yet removed from this file:

https://github.com/dmwm/CMSKubernetes/pull/1409/files/8b2c2ab89b6865d5ded9f1e14591e0c2c2147a88#diff-cc2d7495d3dd0b8fb61cfc89481c74747c3ba277501b1f8270ed096ab42229ee

The push & pull code in the new commit looks good to me, but since the image name has changed I think now the documentation instructions mentioning couchdb alone like:

cmst1@vocms0290:wmagent-couchdb $ docker exec -it couchdb bash
(CouchDB-3.2.2) [cmst1@vocms0290:data]$

would be:

cmst1@vocms0290:wmagent-couchdb $ docker exec -it wmagent-couchdb bash
(CouchDB-3.2.2) [cmst1@vocms0290:data]$

Is that right? Could you please change that to be consistent with the new name?

todor-ivanov · 2024-03-18T22:44:22Z

hi @khurtado, the environment variable I kept there on purpose just to stay consistent with the structure of the rest of the containers. And also to have the placeholder once we start having those database accesses properly set with the correct database level access rights for the accounts (which, IIUC, we are to do it in some near future).

For the command for running/accessing and connecting to the container nothing has changed. The image name has changed, but the container name is still couchdb, so this:

docker exec -it couchdb bash

is still valid.

khurtado

Thank you Todor! I just marked the requests as resolved and approved the PR.

docker/pypi/wmagent-couchdb/manage

khurtado · 2024-03-13T14:43:58Z

docker/pypi/wmagent-couchdb/Dockerfile

+ENV COUCH_LOG_DIR=$COUCH_CURRENT_DIR/logs
+ENV COUCH_DEPLOY_DIR=/usr/local
+ENV COUCH_ENV_FILE=$COUCH_DEPLOY_DIR/deploy/env.sh
+ENV COUCH_SECRETS_FILE=$COUCH_ADMIN_DIR/CouchDB.secrets


If this is var not being used anymore, it can be removed.

This comment is marked as resolved, but I see no follow up. It might be confusing to leave an unused secret reference in this docker file. I would suggest to remove this COUCH_SECRETS_FILE line?

This was discussed with @khurtado in another comment/thread as well ( I cannot remember already which exactly), but we have agreed to keep the variable as a placeholder for the sake of consistency with the MariaDB container and because we do plan to fix this single user problem in the future.

I would leave it commented out, but that's Okay.

todor-ivanov · 2024-03-20T15:31:35Z

thanks @khurtado

khurtado · 2024-03-25T13:49:47Z

@amaltaro Do you want to take a final look at the PR before merging it?

amaltaro · 2024-03-25T14:12:22Z

Yes, I plan to look into it by tomorrow. Apologies for the delay.

todor-ivanov · 2024-03-27T16:39:26Z

@amaltaro would you be able to look into this before the start of this long weekend.
I'd really want to complete this before that.

Thanks in advance

amaltaro

Todor, it's looking good in general, however I did leave many comments/questions/requests along the code.

In addition to that, please update the initial pull request description.

docker/pypi/wmagent-couchdb/README.md

amaltaro · 2024-03-27T18:46:41Z

docker/pypi/wmagent-couchdb/local.ini

+cacert_file = /data/certs/servicecert.pem
+ssl_certificate_max_depth = 10
+verify_ssl_certificates = false
+fail_if_no_peer_cert = false


This configuration is new w.r.t. the actual one. I would drop it unless there is a strong reason to keep it.

amaltaro · 2024-03-27T18:47:11Z

docker/pypi/wmagent-couchdb/local.ini

+cacert_file = /data/certs/servicecert.pem
+ssl_certificate_max_depth = 10
+verify_ssl_certificates = false
+fail_if_no_peer_cert = false


docker/pypi/wmagent-couchdb/local.ini

docker/pypi/wmagent-couchdb/manage

todor-ivanov · 2024-03-28T07:52:29Z

@amaltaro I have addressed all your comments please take another look.

amaltaro

Thank you for the prompt turnaround, Todor. I left a few more comments along the code.

In addition, please add a few more details to the PR description. For someone that didn't work on this, it's not clear where docker images are uploaded; what the assumptions are (secrets? accounts?); current couchdb version; any relevant changes w.r.t. the current couchdb setup.

amaltaro · 2024-03-28T11:35:00Z

docker/pypi/wmagent-couchdb/Dockerfile

+ENV COUCH_LOG_DIR=$COUCH_CURRENT_DIR/logs
+ENV COUCH_DEPLOY_DIR=/usr/local
+ENV COUCH_ENV_FILE=$COUCH_DEPLOY_DIR/deploy/env.sh
+ENV COUCH_SECRETS_FILE=$COUCH_ADMIN_DIR/CouchDB.secrets


This comment is marked as resolved, but I see no follow up. It might be confusing to leave an unused secret reference in this docker file. I would suggest to remove this COUCH_SECRETS_FILE line?

docker/pypi/wmagent-couchdb/README.md

todor-ivanov · 2024-03-28T14:10:40Z

@amaltaro take another look

amaltaro

Thanks Todor, it looks good to me.

todor-ivanov force-pushed the Add_WMAgentCouchDB branch from 2e1a1d2 to 4d68b80 Compare August 21, 2023 13:57

Add minimal CouchDB container for WMAgent

cd32771

Remove anything unnecessary Remove local-upstream-minimal.ini

todor-ivanov force-pushed the Add_WMAgentCouchDB branch from 4d68b80 to cd32771 Compare August 21, 2023 14:00

todor-ivanov added 6 commits August 22, 2023 17:12

Mount config area from host && Read local.ini from host config && Fix…

d00c726

… broken tag build parameter

Fix current link in /data/dockerMount

a34e8f4

Rename COUCH_ROOT_DIR to HOST_MOUNT_DIR

5582cd3

Typo

3f1873e

Fix broken Tls handshake decode error && Downgrade to 3.2.2 && Remove…

04d1a86

… security string checks from manage script.

Add some leftover coments/steps from the wmagent image to be implemen…

1a6b700

…ted here

todor-ivanov mentioned this pull request Feb 6, 2024

Deploy and run WMAgent with Docker container dmwm/WMCore#11314

Open

36 tasks

todor-ivanov requested a review from khurtado February 28, 2024 04:52

todor-ivanov mentioned this pull request Feb 28, 2024

WMAgent: install/run CouchDB from Dockerhub dmwm/WMCore#11312

Closed

Add local.ini file to the docker image

b0fa849

todor-ivanov mentioned this pull request Mar 8, 2024

Tier0 WMAgent deployment model for Alma9/RHEL9 dmwm/WMCore#11890

Closed

Add manage init && Properly parse and load credentials from ${WMA|COU…

7dbe77f

…CH}_SECRETS_FILE && Start couchDB from run.sh

todor-ivanov requested a review from amaltaro March 13, 2024 12:48

Configure a single admin account from WMAGent.secrets file && Parse o…

8b2c2ab

…nly admin section Typo

todor-ivanov force-pushed the Add_WMAgentCouchDB branch from ca5ddbc to 8b2c2ab Compare March 13, 2024 12:52

khurtado reviewed Mar 13, 2024

View reviewed changes

docker/pypi/wmagent-couchdb/README.md Outdated Show resolved Hide resolved

docker/pypi/wmagent-couchdb/couchdb-docker-build.sh Outdated Show resolved Hide resolved

docker/pypi/wmagent-couchdb/couchdb-docker-run.sh Show resolved Hide resolved

Update docker/pypi/wmagent-couchdb/README.md

92977f9

Co-authored-by: Kenyi Hurtado <khurtado@nd.edu>

todor-ivanov requested a review from khurtado March 18, 2024 10:19

Add Push && Pull options to the -run.sh && -build.sh scripts

14e560e

Forgotten extra repository variable

todor-ivanov force-pushed the Add_WMAgentCouchDB branch from decc667 to 14e560e Compare March 18, 2024 10:38

khurtado approved these changes Mar 20, 2024

View reviewed changes

amaltaro requested changes Mar 27, 2024

View reviewed changes

Alan's review comments

50d44bb

todor-ivanov requested a review from amaltaro March 28, 2024 07:52

amaltaro requested changes Mar 28, 2024

View reviewed changes

Alan's review - Second iteration

22a7cce

todor-ivanov requested a review from amaltaro March 28, 2024 14:02

amaltaro approved these changes Mar 28, 2024

View reviewed changes

todor-ivanov merged commit 326de33 into dmwm:master Mar 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add minimal CouchDB container for WMAgent #1409

Add minimal CouchDB container for WMAgent #1409

todor-ivanov commented Aug 18, 2023 •

edited

Loading

khurtado commented Mar 7, 2024

todor-ivanov commented Mar 7, 2024

khurtado commented Mar 7, 2024

todor-ivanov commented Mar 11, 2024 •

edited

Loading

khurtado commented Mar 12, 2024

todor-ivanov commented Mar 13, 2024

todor-ivanov commented Mar 13, 2024

khurtado left a comment •

edited

Loading

khurtado commented Mar 15, 2024

todor-ivanov commented Mar 18, 2024

khurtado commented Mar 18, 2024

todor-ivanov commented Mar 18, 2024 •

edited

Loading

khurtado left a comment •

edited

Loading

khurtado Mar 13, 2024

amaltaro Mar 28, 2024

todor-ivanov Mar 28, 2024

amaltaro Mar 28, 2024

todor-ivanov commented Mar 20, 2024

khurtado commented Mar 25, 2024

amaltaro commented Mar 25, 2024

todor-ivanov commented Mar 27, 2024

amaltaro left a comment

amaltaro Mar 27, 2024

amaltaro Mar 27, 2024

todor-ivanov commented Mar 28, 2024

amaltaro left a comment

amaltaro Mar 28, 2024

todor-ivanov commented Mar 28, 2024

amaltaro left a comment

Add minimal CouchDB container for WMAgent #1409

Add minimal CouchDB container for WMAgent #1409

Conversation

todor-ivanov commented Aug 18, 2023 • edited Loading

khurtado commented Mar 7, 2024

todor-ivanov commented Mar 7, 2024

khurtado commented Mar 7, 2024

todor-ivanov commented Mar 11, 2024 • edited Loading

khurtado commented Mar 12, 2024

todor-ivanov commented Mar 13, 2024

todor-ivanov commented Mar 13, 2024

khurtado left a comment • edited Loading

Choose a reason for hiding this comment

khurtado commented Mar 15, 2024

todor-ivanov commented Mar 18, 2024

khurtado commented Mar 18, 2024

todor-ivanov commented Mar 18, 2024 • edited Loading

khurtado left a comment • edited Loading

Choose a reason for hiding this comment

khurtado Mar 13, 2024

Choose a reason for hiding this comment

amaltaro Mar 28, 2024

Choose a reason for hiding this comment

todor-ivanov Mar 28, 2024

Choose a reason for hiding this comment

amaltaro Mar 28, 2024

Choose a reason for hiding this comment

todor-ivanov commented Mar 20, 2024

khurtado commented Mar 25, 2024

amaltaro commented Mar 25, 2024

todor-ivanov commented Mar 27, 2024

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Mar 27, 2024

Choose a reason for hiding this comment

amaltaro Mar 27, 2024

Choose a reason for hiding this comment

todor-ivanov commented Mar 28, 2024

amaltaro left a comment

Choose a reason for hiding this comment

amaltaro Mar 28, 2024

Choose a reason for hiding this comment

todor-ivanov commented Mar 28, 2024

amaltaro left a comment

Choose a reason for hiding this comment

todor-ivanov commented Aug 18, 2023 •

edited

Loading

todor-ivanov commented Mar 11, 2024 •

edited

Loading

khurtado left a comment •

edited

Loading

todor-ivanov commented Mar 18, 2024 •

edited

Loading

khurtado left a comment •

edited

Loading