THREESCALE-10279 - Use a non-root user for built-in MySQL and PostgreSQL DBs + Secret Update

[valerymo]

Jira: https://issues.redhat.com/browse/THREESCALE-10279
Use a non-root user by default for built-in MySQL and PostgreSQL DBs

• DB_USER and DB_PASSWORD were removed from system-database secret

• It was decided to move this PR/Task to 2.16

• New Jira opened for 2.15, for Documentation update to change Mysql User privilages from Select to ALL - https://issues.redhat.com/browse/THREESCALE-10938

Validation

Retesting 02.Apr

INTERNAL DB

Test	Status	Comments
MySQL - Fresh installation - Local	[valid]	GRANT ALL PRIVILEGES ON system.* TO mysql@% , 3scale tables populated in system db
MySQL - Fresh installation OLM	[valid]	GRANT ALL PRIVILEGES ON system.* TO mysql@% , 3scale tables populated in system db
MySQL - Upgrade from master	[?]	mysql User has All provilages after Initial Installation (master). Upgrade: system-mysql deployment not reconciled. After manual change of secret, and recreate APIM - system-mysql pod is running. system-app-pre pod - completed. Question - we should have system-database Secret and system-mysql Deployment to be updated after 3scale Upgrade. Seems need to implement it - check if Upgrade, and enforce their reconciliation. Is it correct?
Postgres - Fresh installation - Local	[]
Postgres - Fresh installation	[]
Postgres - Upgrade from master	[]

EXTERNAL DB

Test	Status	Comments
MySQL - Fresh installation	[]
MySQL - Upgrade from master	[]
Postgres - Fresh installation	[]
MySQL - Upgrade from master	[]

Previouse validation

What was done for validation (briefly):

• 3scale local installation from branch THREESCALE-10279
• installation from Operators Hub (image)
• Upgrade from master to PR's branch
  • User test1 was created in Initial installation; to check login with this user after upgrade.
  • Upgrade was done successfully
    • logged in with users: test1, mysql, root
    • Note Deleting of system-database secret and system-mysql deploy were required to have mysql pod running with new secret. It should be documented.
    • Detailed information/logs for Upgrade process is below

3scale Upgrade - MySql Test

• clean previouse catalogsource if available

$ oc delete catalogsource threescale-operators --namespace=openshift-marketplace

• Install new catalogsource

kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: threescale-operators
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/vmogilev_rhmi/3scale-index:0.0.1
EOF

• s3-creds-secret.yaml

kind: Secret
apiVersion: v1
metadata: 
  name: s3-credentials
  namespace: 3scale-test
data: 
  AWS_ACCESS_KEY_ID: QUtJxxxxx
  AWS_SECRET_ACCESS_KEY: aU5Vxxxxxx
  AWS_BUCKET: dm1vxxxxx
  AWS_REGION: ZXUtxxxxx
type: Opaque

• apimanagerCR.yaml

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: apps.xxxxxx.axxxx.xxx.xxxxx.org

Check Initial 3scale installation

$ oc get secret system-database -oyaml

apiVersion: v1
data:
  DB_PASSWORD: bE5Saxxx
  DB_USER: bXlxxx
  URL: bXlzcWwxxxxx
kind: Secret
metadata:
 ......
type: Opaque

=============

• DB_USER:
  $ echo bXlzcWw= |base64 -d
  mysql

• URL:
  $ echo bXlzcWwyOixxxx |base64 -d
  mysql2://root:wxxxxx0@system-mysql/system

• Login to mysql as root:

sh-4.2$ mysql -u root
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 435
.....
mysql> 

mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
| system             |
+--------------------+
5 rows in set (0.00 sec)


mysql> use mysql;
Database changed
mysql> select user();
+----------------+
| user()         |
+----------------+
| root@localhost |
+----------------+
1 row in set (0.00 sec)

mysql> select user, host, User_attributes from user;
+------------------+-----------+-----------------+
| user             | host      | User_attributes |
+------------------+-----------+-----------------+
| mysql            | %         | NULL            |
| root             | %         | NULL            |
| mysql.infoschema | localhost | NULL            |.....
6 rows in set (0.00 sec)

mysql> SHOW COLUMNS FROM user;
.....

mysql> 

- login with mysql user
(password is taken from secret -  DB_PASSWORD)

sh-4.2$ mysql -u mysql -p         
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
.....
mysql> show databases; 
+--------------------+
| Database           |
+--------------------+
| information_schema |
| system             |
+--------------------+
2 rows in set (0.01 sec)

mysql> 
mysql> use system;
Database changed
mysql> select user();
+-----------------+
| user()          |
+-----------------+
| mysql@localhost |
+-----------------+
1 row in set (0.00 sec)

mysql> select database();
+------------+
| database() |
+------------+
| system     |
+------------+
1 row in set (0.00 sec)

-------------------

Add user test1 to initial installation
We will try login with this user after upgrade

mysql> select user from user;
+------------------+
| user             |
+------------------+
| mysql            |
| root             |
......
6 rows in set (0.00 sec)

mysql> CREATE USER 'test1'@'localhost' IDENTIFIED BY 'test1';
Query OK, 0 rows affected (0.01 sec)

mysql> select user from user;
+------------------+
| user             |
+------------------+
| mysql            |
| root             |
.......
| test1            |
+------------------+
7 rows in set (0.00 sec)

*** UPGRADE 3scale ****

• Update Catalog source - upgrade will be executed

• Note that mysql deployment and secret are not updated after Upgrade:

$ oc rollout history deployment.apps/system-mysql
deployment.apps/system-mysql
REVISION CHANGE-CAUSE
1

$ oc get secret system-database -oyaml
apiVersion: v1
data:
DB_PASSWORD: ZzJMaxxx
DB_USER: bXlxxx
URL: bXlzcWwyOixxxxx
kind: Secret
.....
type: Opaque

$ oc get pod |grep mysql
system-mysql-d7dbf74c7-qz6vg 1/1 Running 0 13m

$ oc rsh system-mysql-d7dbf74c7-qz6vg
sh-4.2$ mysql -u root
Welcome....
....
mysql> use mysql;
Database changed
mysql> select user from user;
+------------------+
| user |
+------------------+
| mysql |
| root |
.....
| test1 |
+------------------+
7 rows in set (0.00 sec)

mysql>

---

• Delete secret and delete deployment
  Notes - it was found that not enough to rollour reployment (kubectl rollout restart deployment/system-mysql), as pod had error CreateContainerConfigError

$ oc delete  deployment/system-mysql
deployment.apps "system-mysql" deleted

$ oc get secret system-database
NAME              TYPE     DATA   AGE
system-database   Opaque   1      52s
$ oc get secret system-database -oyaml
apiVersion: v1
data:
  URL: bXlzcWwyOi8vbXlxxxxxx
kind: Secret
metadata:
.....
type: Opaque

$ oc get pod |grep mysql
system-mysql-8c88b67f6-pq7vf                                 1/1     Running     0             53s

• Login to MySql with test1 user

$ oc rsh system-mysql-8c88b67f6-pq7vf
sh-4.2$ mysql -u test1 -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
...
mysql> 
mysql> select user();
+-----------------+
| user()          |
+-----------------+
| test1@localhost |
+-----------------+
1 row in set (0.00 sec)

• login with root user
  NO Password

sh-4.2$ mysql -u root    
Welcome to the MySQL monitor.  Commands end with ; or \g.
...

• loging with mysql user
  NOTE - Password in mysql user changed, it's differ from initial installation.
  Passwork should be taken from URL in secret:

$ echo bXlzcWwyOi8vbXlzcWwxxxxxxxx== |base64 -d
mysql2://mysql:1Wxxxxb@system-mysql/system

login:

sh-4.2$ mysql -u mysql -p
Enter password: 
Welcome to the MySQL monitor.  Comma
...

NOTE. Update of deployment and secret - required. Update order:

1. Delete deployment : oc delete deploy system-mysql
2. Updated Secret: oc edit secret system-database : delete fields DB_USER and DB_PASSWOR

3scale Upgrade - Postgres Test

• CatalogSource

kubectl apply -f - <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: threescale-operators
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/vmogilev_rhmi/3scale-index:0.0.1
EOF

• APIManager CR

piVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  resourceRequirementsEnabled: false
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
    database:
      postgresql:
          priorityClassName: system-node-critical     
  wildcardDomain: apps.vmo01.2td0.s1.devshift.org

• Connect to postgres

$ oc rsh system-postgresql-xxxxxx

sh-4.2$ psql -d system -U system
psql (10.17)
Type "help" for help.

system=>
system=> \du
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 system    |                                                            | {}

system=> \q

sh-4.2$ psql -d system -U postgres
 
system=# SELECT CURRENT_ROLE;
 current_role 
--------------
 postgres
(1 row)

system=# CREATE USER test1 WITH PASSWORD 'test1';
system=# CREATE USER test2 WITH PASSWORD 'test2';

system=# \du 
 <pre>ystem=# \du 
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 system    |                                                            | {}
 test1     |                                                            | {}
 test2     |                                                            | {}

system=# \q
sh-4.2$ exit

Upgrade

• update catalogsource

oc edit catalogsource threescale-operators --namespace=openshift-marketplace

• wait for upgrade completed (check UI, or apimanager)

$ oc describe apimanager example-apimanager |grep " Status:"
    Status:                True

• check secret and deploy / pod

  • secret and pod didn't change after upgrade

• delete deployment:

 oc delete deploy system-postgresql

results:
- secret - not changed;
URL (after decreption): postgresql://system:XXXXXXy@system-postgresql/systembase64:
Notes Behavior is differ than for MySql, becasue Postgres used "system" user, and no change of this user in PR
- pod is recreated successfully

• check DB:

system-postgresql-55d67dd677-9p2qz                           1/1     Running     0             2m5s
[vmogilev@vmogilev 3scale-operator] (THREESCALE-10279)$ rsh coc rsh ^C
[vmogilev@vmogilev 3scale-operator] (THREESCALE-10279)$ oc rsh system-postgresql-55d67dd677-9p2qz
sh-4.2$ psql -d system -U postgres
psql (10.17)
Type "help" for help.

system=# \du
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 system    |                                                            | {}
 test1     |                                                            | {}
 test2     |                                                            | {}

• delete not used fields (DB_USER and DB_PASSWOR) from secret:

$ oc get secret system-database -oyaml
apiVersion: v1
data:
  URL: cG9zdGdyZXNxbDovL3N5c3RlbTpSUkxvTjhyeUBzeXN0ZW0tcG9zdGdyZXNxbC9zeXN0ZW0=
kind: Secret
metadata:
.....

• check that pod will run after recreation

$ oc delete pod system-postgresql-55d67dd677-9p2qz
pod "system-postgresql-55d67dd677-9p2qz" deleted
$ oc get pod |grep postg
system-postgresql-55d67dd677-6w4jv                           1/1     Running     0             81s

sh-4.2$ psql -d system -U test1
psql (10.17)
Type "help" for help.

system=> \du
                                   List of roles
 Role name |                         Attributes                         | Member of 
-----------+------------------------------------------------------------+-----------
 postgres  | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
 system    |                                                            | {}
 test1     |                                                            | {}
 test2     |                                                            | {}

system=> 

NOTE. Update of deployment and secret - required. Update order:

1. Delete deployment : oc delete deploy system-postgresql
2. Updated Secret: oc edit secret system-database : delete fields DB_USER and DB_PASSWOR

NOTEs for Upgrade. Update of deployment and secret - required after Upgrade. Update order:

1. Delete deployment :
   • for Postgres: oc delete deploy system-postgresql
   • for MySql: oc delete deploy system-mysql
2. Update Secret: oc edit secret system-database : delete fields DB_USER and DB_PASSWOR

External MySql DB Validation

To Create external Mysql db for Test

• create project mysql-test

• run:
  oc new-app -e MYSQL_USER=admin -e MYSQL_PASSWORD=12345 -e MYSQL_DATABASE=system registry.redhat.io/rhscl/mysql-56-rhel7

• Secret:

-------  secret was --
apiVersion: v1
kind: Secret
metadata:
  creationTimestamp: null
  name: system-database
  namespace: 3scale-test
stringData:
  URL: mysql2://test1:12345@<HOST>/mysql
type: Opaque

• Apimanager CR:

apiVersion: apps.3scale.net/v1alpha1
kind: APIManager
metadata:
  name: example-apimanager
spec:
  externalComponents:
    system:
      database: true
  system: 
    fileStorage: 
      simpleStorageService: 
        configurationSecretRef: 
          name: s3-credentials
  wildcardDomain: <wildcardDomain>```

Grant limited permission to user and check remote connection

mysql> show grants for test1;
+------------------------------------------------------------------------------------------------------+
| Grants for test1@% |
+------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON . TO 'test1'@'%' IDENTIFIED BY PASSWORD '00A51F3F48415C7D4E8908980D443C29C69B60C9' |
| GRANT SELECT, INSERT, UPDATE ON mysql. TO 'test1'@'%' |
+------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Result: Error in system-app-pre job - Connected but can't create mysql (?? trying to create)

Need provide ALL PRIVILEGES to user to allow remote connection and operation
NOTES Please note that ALL PRIVILEGES for User on specific DB - it's not the same as Root privilages.

Grant ALL PRIVILEGES ON mysql DB for test user

mysql> GRANT ALL PRIVILEGES ON mysql.* to test1;
Query OK, 0 rows affected (0.00 sec)

mysql> show grants for test1;
+------------------------------------------------------------------------------------------------------+
| Grants for test1@% |
+------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON . TO 'test1'@'%' IDENTIFIED BY PASSWORD '00A51F3F48415C7D4E8908980D443C29C69B60C9' |
| GRANT ALL PRIVILEGES ON mysql. TO 'test1'@'%' |
+------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

Result - Successfull. Connected and 3scale tables created

.....
Creating scope :admins. Overwriting existing method User.admins.
Creating scope :by_name. Overwriting existing method Cinstance.by_name.
Creating scope :provided_by. Overwriting existing method ServiceContract.provided_by.
[core] non-native log levels verbose, notice, critical emulated using UNKNOWN severity
Backend Internal API version 3.4.3 status: ok
Connected to mysql2://test1@<HOST>/mysql
Connected to redis://system-redis:6379/1

mysql> show tables;
+--------------------------------+
| Tables_in_mysql                |
+--------------------------------+
....
| oidc_configurations            |
....
+--------------------------------+
117 rows in set (0.00 sec)
mysql> 

Check if User and DB are not exist

change URL:
URL: mysql2://test2:12345@mysql-56-rhel7.mysql-test.svc.cluster.local/testdb2

Mysql2::Error::ConnectionError: Access denied for user 'test2'@'10.130.2.41' (using password: YES)

mysql> SELECT create_time
-> FROM information_schema.tables
-> WHERE table_schema = 'mysql'
-> AND table_name = 'oidc_configurations';
+---------------------+
| create_time |
+---------------------+
| 2024-03-26 07:18:32 |
+---------------------+
1 row in set (0.01 sec)

mysql>

Recheck with one more User

mysql> create user test2 identified by '12345';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON mysql.* to test2;
Query OK, 0 rows affected (0.00 sec)

URL in secret
URL: mysql2://test2:12345@mysql-56-rhel7.mysql-test.svc.cluster.local/mysql

Job logs

• success, similar to test1 user

....
Backend Internal API version 3.4.3 status: ok
Connected to mysql2://test2@mysql-56-rhel7.mysql-test.svc.cluster.local/mysql
Connected to redis://system-redis:6379/1

mysql> SELECT create_time FROM information_schema.tables WHERE table_schema = 'mysql' AND table_name = 'oidc_configurations';
+---------------------+
| create_time         |
+---------------------+
| 2024-03-26 07:18:32 |
+---------------------+
1 row in set (0.00 sec)

Check if any table was updated after user test2 connection

SELECT 
    TABLE_NAME,
    UPDATE_TIME
FROM 
    INFORMATION_SCHEMA.TABLES
WHERE 
    TABLE_SCHEMA = 'mysql'
    AND UPDATE_TIME > '2024-03-26 07:18:32';

mysql> SELECT      TABLE_NAME,     UPDATE_TIME FROM      INFORMATION_SCHEMA.TABLES WHERE      TABLE_SCHEMA = 'mysql'     AND UPDATE_TIME > '2024-03-26 07:18:32';
+------------+---------------------+
| TABLE_NAME | UPDATE_TIME         |
+------------+---------------------+
| db         | 2024-03-26 08:11:59 |
| proc       | 2024-03-26 07:18:33 |
| user       | 2024-03-26 08:11:00 |
+------------+---------------------+
3 rows in set (0.00 sec)

We can see that No 3scale tables were updated after second user connection

Summary

• User require ALL PRIVILEGES Grants to connect to remote MySQL
• We suppose that user already exists and DB already exists
• system-database secret created by user/customer, and contains URL similar to below:
  URL: mysql2://test2:test2password@<HOST>/mysql
  • system-database secret does Not contains absolete fields DB_USER and DB_PASSWORD.

Notes for External MySQL

• User require ALL PRIVILEGES Grants to connect to remote MySQL
• In Validation we supposed that user already exists and DB already exists
• system-database secret created by user/customer, and contains URL similar to below:
  URL: mysql2://test2:test2password@<HOST>/mysql
  • system-database secret does Not contains absolete fields DB_USER and DB_PASSWORD.

[carlkyrillos]

I left a few comments but I was unable to complete the verification steps because the installation never completed for both fresh install and upgrade scenarios. Something is blocking the apicast-production Deployment from becoming healthy (this is also what caused the e2e test to fail). I'll continue to debug and will report back if I find the root cause.

[valerymo]

I left a few comments but I was unable to complete the verification steps because the installation never completed for both fresh install and upgrade scenarios. Something is blocking the apicast-production Deployment from becoming healthy (this is also what caused the e2e test to fail). I'll continue to debug and will report back if I find the root cause.

Thank you for Review @carlkyrillos . Hope I addressed your comments. Also - I removed Root password (I asked in chat for people openian, waiting, but I see that it's not in use). Also I added Validation logs and notes for External MySql test. Thank you

[valerymo]

@MStokluska , thanks very much for external Mysql process explanation. I retested and added logs and notes to Validation section.

[valerymo]

• TODO: External Postgres test
• TODO: Secret and Deployment reconciliation checking and improvement for Internal DB

[valerymo]

Looks like requirements changed. Waiting.
cc @carlkyrillos , @MStokluska
Thank you

[valerymo]

/retest

[openshift-ci]

@valerymo: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/test-e2e	350e790	link	true	/test test-e2e

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.