Signing scripts don't seem to be mounted to relevant pods

[mgoldenberg]

Version
v1.0.0-beta.4

Describe the bug

After deploying a Pulp instance with spec.signing_secret and spec.signing_scripts properly set, adding content to a signed RPM repository fails due to the following error.

[Errno 2] No such file or directory: '/var/lib/pulp/scripts/collection_script.sh'

I can confirm that my signing scripts are present in the database by querying through the command line tool.

$ pulp signing-service list
[
  {
    "pulp_href": "/pulp/api/v3/signing-services/018eaf6c-8f38-7ef9-aa60-a9c03f8d0775/",
    "pulp_created": "2024-04-05T18:01:51.529782Z",
    "pulp_last_updated": "2024-04-05T18:01:51.529827Z",
    "name": "container-signing-service",
    "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBGYOx+IBEADr+2TtnsJhq7pgGn2V4Uk3kM5rdMe4PKBVR3DoW21L4dzVxlFZ\nldxHP/1N8ftY8MMxquoXJZhkBy3tkjC20MveuAEvdxJSX09upb1Gz8tfYejDf/EM\nDf9/RWP9FgM2if60UhsHycif+tcsxGNY48ARYGdmFpVow7CY1NhRi1oakgZDmlX3\n7FvjWEGZ46Yff4sUxoqjEo8moTMKzeK9Z/G/sjuxD/hpMYM0Tj2LZzWoB5E1cjiR\neIayKZx3wJR8obxOsfpxG05yM0P1DTEM3tY8Qte31DKGFh2FJ3zNMBH00r/jaB6Z\nj3q/Gn2bihShACzNqZsNADgb7pcjLAGuNJ4zPuHhlLCkI16BYB7u6lJDuZAkvTvn\nL+4SpgmO9I0DVDIHrknJOfi5S1DFkLtLs3FCEQVzrQ3aOsEg2eA92eiM3WRomK27\nrRdGiQk2MtL02cqv2b9LNlIZWKcLRtO3Pr9D3VmVPBcDyVXkSFcI43OnfsuMeJZG\nwMv9jUtqBSKdKTnyQQAFsqTU9Qw0mllikJfF3iUYby1FzUHNHN7R9U8dY+SU53Xd\nkrPE1y24mwbqrHhZGM+DPOPF43TJZEKy9J0zNDQSc0jLOa9tgBrhmRIh4rY7pYnP\noe0Zf79jui/6zSp84ctt5Jjsjea/HTw04Qvm9mr7fBlZWjmYkcTgypTnoQARAQAB\ntCVGT0FNIEFydGlmYWN0cyA8YXJ0aWZhY3RzQGZvYW0uc3BhY2U+iQJOBBMBCgA4\nFiEEpiEmZyQv7oEpXQ2NEIdUzv9Hvl0FAmYOx+ICGwMFCwkIBwIGFQoJCAsCBBYC\nAwECHgECF4AACgkQEIdUzv9Hvl1t+Q//VdiiSTMiSs6KKNI/xarg3Dq3QltuFYQp\nVhoOTAuPDt8S3eh0VBwXPW8vleipRVp5xYiKlhG7yltiinz+4zSMK4HPuM05q7JD\nB26MANF9T5ejnQLIamwwQh7jMWqpIT5G7rU9Mw8+5rJHtLPjsp4PRsvo66vVxYPJ\ntd36Pm+MVbIhKBmoP/QI8sJDyiz6DwD2ylSu2PeUIbohqZeRaKt6MlTLiIpFjQEN\nWT+HdfbkIU2l4uU3SVApT2GNNKE+VrPJx4mDAEJvcktUK3liqIN2xV3RraPCPcEE\n7reYufKUZxCQ8YD2KnnmGQBqscqYS8iCb7t2ysZrZzoGUXyKJMdJMmcsyPx8O2Us\nnZc9PyifIOJrtNceR+lgYNC7KJoqWPkeF8Fk4qlL+2SEqK8uDXEyNc6gmJtj8BFw\n2Sc3WaPZymYnGC8hWW2uW1PDk72OQVZOnbKnhMZ9EPChU97VFa1DZFS6I4dSS6pb\nc2FYB+q1HKdhBSrpz7GhQg3FF9KRI/1PQeqD2TgGaFgRfuqCOtPP7Gj1vFiLXuU+\nqDxQBrZBEW7VfliFWECknR3rKGPPXyYOQY0cKsrI8hfScPmpPzykksEsPseLlMrj\nODFohefzeof3VvYIYsnOapiZUF7USqJALyntcZiBcS/M9pWECVEwgafukN/E3zBT\nkx1DP9j0fvq5Ag0EZg7H4gEQALHk7T18pk04kBjS7zeJndjtBOA336Hgyfnp+Hz4\n7EttbaaLqwdaXNJXgP6KBXbhGb81V3nURihbyo3AcyM71iNfYHFw2A5IuNX/r8C5\nirLzLbchwNX+sdEwugDHJy7zwh6XhlGc3aCZTb/yvo2mkwWt8n/Vb+KoyHiFNJwr\nHoxv6He/gs2ZX5A+g1jTliMy1YuVv+XR4LJ1QWW/lcKJIb4o7nIhEt0UoH4NWpHV\nhf4vh5gtXoCiqHVAz+xfeWVyPKTk+A54eEZTRe+tV7BsBZvxv2nQTpz0+Mo+Il/K\ncHDGvH2lQVULVWThn9vluMZqgYFvH5OLtSVPoNOkGct3xjOwraRf0kg/CEIw/Bh9\nURK94rgZHAQrdIHE65PI90JjdRKbNrW1zxGnU/pYblajq5Lojqf66BJY5Fpj5wCd\nxQZLJRLysM7ITGpPbvK+FeJ/+WzrTlyuGC9ejktY6FyPnWjTEBPjebalxMFvy7cY\ntq9MXC1u2PJj8vMYGmwEkTvw1jpNJuzV3nFJetW5kz5stM8Mgmp6jF6OekOfxifi\nmdH/zdSjMWQAOeXfDzmAgudQVOwDKatiLly/9gTziWFZU+TyTuS/kxoIaOonbfH1\ncEx+F++BzxqlMHJQIJ/xgfqvJMoAxqny8sTNDPh7z+73USMXy01opoWBubCAWjp0\n3BvxABEBAAGJAjYEGAEKACAWIQSmISZnJC/ugSldDY0Qh1TO/0e+XQUCZg7H4gIb\nDAAKCRAQh1TO/0e+XXhvEACDpBvXEn+x60QMNRvuuKRUHOuzWjg7vd0SD81OAr6J\nukd79/cMTrnunJEo9MYFmUdbbTDtzkRzxvPKc+HqrkCDQ8ef0DMjFbfSydLOIA4a\n2zBLXPUf9KxsIkv3QBY4i9i/q+Tz2Mn8p4K/NrPxN3EFTj1grfIdesWBEe0kZ1vx\nchveoCNomBfI+vZBx2S/Q+jSiMt1EQ42OsdWq8Sb6vdAhaeuHOjB+7xc7CHXAzeY\nC2MzWy1pKWZldCojPOePAX+Lnhm+y26duxWu4Ld4813GmJLvWovGzVIyOglNbVuV\naH9CK5NwM1KxE7rtfm+JIaU6vw7vLzooB9u6GvZtwzIwQvNWsfMtvfKxf9NAyznh\nGHOg/ZJm7u0Bblws/JZt2nQxjZAngk0CLpCAHY5Axa7V+iDCJO03+lU/if7TCTsD\n3T86HC1Vwc0DY6V950Ja842xBWNji02wqbjhvQpnvxZ6QCqKH1s26aX5TX97RAgL\n41bSQVhYV8fxipjTFfvWFX11/4cojGC+556i4VzSA8A/Oxe3HbAK67OJ3jQfPoZk\nYqUhqknELD4FxmjiR8kAqUfsV8SndmkQnDlUerVImqicc9YxI86kgOvfoQeS19A6\niCAGxPpV3PzkoEDBCYBAF5talSn8tSnTzGsiS2ZZcyoh6OyZplJq3ncSABAIfws3\n9w==\n=Ldaz\n-----END PGP PUBLIC KEY BLOCK-----\n",
    "pubkey_fingerprint": "A6212667242FEE81295D0D8D108754CEFF47BE5D",
    "script": "/var/lib/pulp/scripts/container_script.sh"
  },
  {
    "pulp_href": "/pulp/api/v3/signing-services/018eaf6c-57f0-776c-8681-3774034a48c1/",
    "pulp_created": "2024-04-05T18:01:37.700008Z",
    "pulp_last_updated": "2024-04-05T18:01:37.700047Z",
    "name": "collection-signing-service",
    "public_key": "-----BEGIN PGP PUBLIC KEY BLOCK-----\n\nmQINBGYOx+IBEADr+2TtnsJhq7pgGn2V4Uk3kM5rdMe4PKBVR3DoW21L4dzVxlFZ\nldxHP/1N8ftY8MMxquoXJZhkBy3tkjC20MveuAEvdxJSX09upb1Gz8tfYejDf/EM\nDf9/RWP9FgM2if60UhsHycif+tcsxGNY48ARYGdmFpVow7CY1NhRi1oakgZDmlX3\n7FvjWEGZ46Yff4sUxoqjEo8moTMKzeK9Z/G/sjuxD/hpMYM0Tj2LZzWoB5E1cjiR\neIayKZx3wJR8obxOsfpxG05yM0P1DTEM3tY8Qte31DKGFh2FJ3zNMBH00r/jaB6Z\nj3q/Gn2bihShACzNqZsNADgb7pcjLAGuNJ4zPuHhlLCkI16BYB7u6lJDuZAkvTvn\nL+4SpgmO9I0DVDIHrknJOfi5S1DFkLtLs3FCEQVzrQ3aOsEg2eA92eiM3WRomK27\nrRdGiQk2MtL02cqv2b9LNlIZWKcLRtO3Pr9D3VmVPBcDyVXkSFcI43OnfsuMeJZG\nwMv9jUtqBSKdKTnyQQAFsqTU9Qw0mllikJfF3iUYby1FzUHNHN7R9U8dY+SU53Xd\nkrPE1y24mwbqrHhZGM+DPOPF43TJZEKy9J0zNDQSc0jLOa9tgBrhmRIh4rY7pYnP\noe0Zf79jui/6zSp84ctt5Jjsjea/HTw04Qvm9mr7fBlZWjmYkcTgypTnoQARAQAB\ntCVGT0FNIEFydGlmYWN0cyA8YXJ0aWZhY3RzQGZvYW0uc3BhY2U+iQJOBBMBCgA4\nFiEEpiEmZyQv7oEpXQ2NEIdUzv9Hvl0FAmYOx+ICGwMFCwkIBwIGFQoJCAsCBBYC\nAwECHgECF4AACgkQEIdUzv9Hvl1t+Q//VdiiSTMiSs6KKNI/xarg3Dq3QltuFYQp\nVhoOTAuPDt8S3eh0VBwXPW8vleipRVp5xYiKlhG7yltiinz+4zSMK4HPuM05q7JD\nB26MANF9T5ejnQLIamwwQh7jMWqpIT5G7rU9Mw8+5rJHtLPjsp4PRsvo66vVxYPJ\ntd36Pm+MVbIhKBmoP/QI8sJDyiz6DwD2ylSu2PeUIbohqZeRaKt6MlTLiIpFjQEN\nWT+HdfbkIU2l4uU3SVApT2GNNKE+VrPJx4mDAEJvcktUK3liqIN2xV3RraPCPcEE\n7reYufKUZxCQ8YD2KnnmGQBqscqYS8iCb7t2ysZrZzoGUXyKJMdJMmcsyPx8O2Us\nnZc9PyifIOJrtNceR+lgYNC7KJoqWPkeF8Fk4qlL+2SEqK8uDXEyNc6gmJtj8BFw\n2Sc3WaPZymYnGC8hWW2uW1PDk72OQVZOnbKnhMZ9EPChU97VFa1DZFS6I4dSS6pb\nc2FYB+q1HKdhBSrpz7GhQg3FF9KRI/1PQeqD2TgGaFgRfuqCOtPP7Gj1vFiLXuU+\nqDxQBrZBEW7VfliFWECknR3rKGPPXyYOQY0cKsrI8hfScPmpPzykksEsPseLlMrj\nODFohefzeof3VvYIYsnOapiZUF7USqJALyntcZiBcS/M9pWECVEwgafukN/E3zBT\nkx1DP9j0fvq5Ag0EZg7H4gEQALHk7T18pk04kBjS7zeJndjtBOA336Hgyfnp+Hz4\n7EttbaaLqwdaXNJXgP6KBXbhGb81V3nURihbyo3AcyM71iNfYHFw2A5IuNX/r8C5\nirLzLbchwNX+sdEwugDHJy7zwh6XhlGc3aCZTb/yvo2mkwWt8n/Vb+KoyHiFNJwr\nHoxv6He/gs2ZX5A+g1jTliMy1YuVv+XR4LJ1QWW/lcKJIb4o7nIhEt0UoH4NWpHV\nhf4vh5gtXoCiqHVAz+xfeWVyPKTk+A54eEZTRe+tV7BsBZvxv2nQTpz0+Mo+Il/K\ncHDGvH2lQVULVWThn9vluMZqgYFvH5OLtSVPoNOkGct3xjOwraRf0kg/CEIw/Bh9\nURK94rgZHAQrdIHE65PI90JjdRKbNrW1zxGnU/pYblajq5Lojqf66BJY5Fpj5wCd\nxQZLJRLysM7ITGpPbvK+FeJ/+WzrTlyuGC9ejktY6FyPnWjTEBPjebalxMFvy7cY\ntq9MXC1u2PJj8vMYGmwEkTvw1jpNJuzV3nFJetW5kz5stM8Mgmp6jF6OekOfxifi\nmdH/zdSjMWQAOeXfDzmAgudQVOwDKatiLly/9gTziWFZU+TyTuS/kxoIaOonbfH1\ncEx+F++BzxqlMHJQIJ/xgfqvJMoAxqny8sTNDPh7z+73USMXy01opoWBubCAWjp0\n3BvxABEBAAGJAjYEGAEKACAWIQSmISZnJC/ugSldDY0Qh1TO/0e+XQUCZg7H4gIb\nDAAKCRAQh1TO/0e+XXhvEACDpBvXEn+x60QMNRvuuKRUHOuzWjg7vd0SD81OAr6J\nukd79/cMTrnunJEo9MYFmUdbbTDtzkRzxvPKc+HqrkCDQ8ef0DMjFbfSydLOIA4a\n2zBLXPUf9KxsIkv3QBY4i9i/q+Tz2Mn8p4K/NrPxN3EFTj1grfIdesWBEe0kZ1vx\nchveoCNomBfI+vZBx2S/Q+jSiMt1EQ42OsdWq8Sb6vdAhaeuHOjB+7xc7CHXAzeY\nC2MzWy1pKWZldCojPOePAX+Lnhm+y26duxWu4Ld4813GmJLvWovGzVIyOglNbVuV\naH9CK5NwM1KxE7rtfm+JIaU6vw7vLzooB9u6GvZtwzIwQvNWsfMtvfKxf9NAyznh\nGHOg/ZJm7u0Bblws/JZt2nQxjZAngk0CLpCAHY5Axa7V+iDCJO03+lU/if7TCTsD\n3T86HC1Vwc0DY6V950Ja842xBWNji02wqbjhvQpnvxZ6QCqKH1s26aX5TX97RAgL\n41bSQVhYV8fxipjTFfvWFX11/4cojGC+556i4VzSA8A/Oxe3HbAK67OJ3jQfPoZk\nYqUhqknELD4FxmjiR8kAqUfsV8SndmkQnDlUerVImqicc9YxI86kgOvfoQeS19A6\niCAGxPpV3PzkoEDBCYBAF5talSn8tSnTzGsiS2ZZcyoh6OyZplJq3ncSABAIfws3\n9w==\n=Ldaz\n-----END PGP PUBLIC KEY BLOCK-----\n",
    "pubkey_fingerprint": "A6212667242FEE81295D0D8D108754CEFF47BE5D",
    "script": "/var/lib/pulp/scripts/collection_script.sh"
  }
]

But the scripts themselves don't seem to be mounted onto the relevant pods. For instance, on the pulp-worker pod, I see the following logs.

pulp [236799e106284e8cb2b638d3851c6d13]: pulpcore.tasking.tasks:INFO: Task 018eafe8-af72-7b77-9894-670c66058700 failed ([Errno 2] No such file or directory: '/var/lib/pulp/scripts/collection_script.sh')
pulp [236799e106284e8cb2b638d3851c6d13]: pulpcore.tasking.tasks:INFO:   File "/usr/local/lib/python3.9/site-packages/pulpcore/tasking/tasks.py", line 70, in _execute_task
    result = func(*args, **kwargs)

  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/tasks/repository.py", line 238, in add_and_remove
    new_version.add_content(models.Content.objects.filter(pk__in=add_content_units))

  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/repository.py", line 1231, in __exit__
    repository.on_new_version(self)

  File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/models/repository.py", line 249, in on_new_version
    tasks.publish(

  File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 385, in publish
    generate_repo_metadata(

  File "/usr/local/lib/python3.9/site-packages/pulp_rpm/app/tasks/publishing.py", line 696, in generate_repo_metadata
    sign_results = signing_service.sign(repomd_path)

  File "/usr/local/lib/python3.9/site-packages/pulpcore/app/models/content.py", line 805, in sign
    completed_process = subprocess.run(

  File "/usr/lib64/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:

  File "/usr/lib64/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,

  File "/usr/lib64/python3.9/subprocess.py", line 1837, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)

And describing the pod does not show any mounts on /var/lib/pulp/scripts.

Containers:
  worker:
    Container ID:  docker://41f0d807eed7aeecab45933fb95f6b902bce6032ecd03793e6f1133950d358bf
    Image:         quay.io/pulp/pulp-minimal:stable
    Image ID:      docker-pullable://quay.io/pulp/pulp-minimal@sha256:d3d0684bc41466762fb3834b1b46d322a19c88b5f519ff9b735752c9281cb406
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/pulp-worker
    State:          Running
      Started:      Fri, 05 Apr 2024 19:21:36 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      exec [/usr/bin/wait_on_postgres.py] delay=3s timeout=10s period=10s #success=1 #failure=1
    Environment:
      POSTGRES_SERVICE_HOST:         <set to the key 'POSTGRES_HOST' in secret 'pulp-pg'>  Optional: false
      POSTGRES_SERVICE_PORT:         <set to the key 'POSTGRES_PORT' in secret 'pulp-pg'>  Optional: false
      PULP_SIGNING_KEY_FINGERPRINT:  A6212667242FEE81295D0D8D108754CEFF47BE5D
      HOME:                          /var/lib/pulp
    Mounts:
      /.ansible/tmp from pulp-ansible-tmp (rw)
      /etc/pulp/keys/database_fields.symmetric.key from pulp-db-fields-encryption (ro,path="database_fields.symmetric.key")
      /etc/pulp/settings.py from pulp-server (ro,path="settings.py")
      /var/lib/pulp/.gnupg from ephemeral-gpg (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9qwkq (ro)

To Reproduce

1. Deploy Pulp instance with spec.signing_secret and spec.signing_scripts set according to metadata signing documentation.
2. Create RPM repository with metadata-signing-service set to container-signing-service.
3. Add content to the RPM repository.

Expected behavior

The content should be added to the RPM repository and the repository should be signed without error.

Additional context

None