Fixes #27398 - enable ansible collection smart proxy content sync #8974

jjeffers · 2020-10-01T19:30:32Z

This PR enables ansible collection content syncing with remote proxies.

There are a few pre-requisites to this PR.

First is to have pulp_ansible v0.4.1 installed on both the the katello host and the remote proxy.
Both the katello host and the remote proxy must have support displayed for ansible_collection under the Pulpcore section in the Smart Proxies -> Services tab.

In the Foreman config/settings.plugins.d/katello.yml on the katello/pulp primary ensure that:
:ansible_collection: true
is present in the content types.

To support the remote proxy with reverse proxy mapping to pulpcore on the katello host, a http conf should be created on the katello/pulp primary, similar to:

ProxyPass /pulp_ansible/galaxy/ http://127.0.0.1:24817/pulp_ansible/galaxy/ retry=0
ProxyPassReverse /pulp_ansible/galaxy/ http://127.0.0.1:24817/pulp_ansible/galaxy/

ProxyPass /pulp/content/ http://localhost:24816/pulp/content/
ProxyPassReverse /pulp/content/ http://localhost:24816/pulp/content/

Also update the /etc/httpd/conf.d/05-foreman-ssl.conf with:

## Proxy rules
  ...
  ProxyPass /pulp_ansible !
  ...

Create a new product and repository of the ansible_collection type, using the galaxy api url such as https://galaxy-dev.ansible.com/api/v2/collections/testing/ansible_testing_content.

Sync that repository. You should now have content available on the katello host (but not the remote proxy... yet).

You can check that the http configuration is correct by issuing an HTTP request for that content via the pulpcore galaxy API:
curl https://centos7-katello-devel-stable.example.com/pulp_ansible/galaxy/Default_Organization/Library/custom/test/test-ansible/api/v2/collections/
And the result should be:

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "name": "ansible_testing_content",
      "namespace": "testing",
      "href": "https://centos7-katello-devel-stable.example.com/pulp_ansible/galaxy/Default_Organization/Library/custom/test/test-ansible/api/v2/collections/testing/ansible_testing_content/",
      "versions_url": "https://centos7-katello-devel-stable.example.com/pulp_ansible/galaxy/Default_Organization/Library/custom/test/test-ansible/api/v2/collections/testing/ansible_testing_content/versions/"
    }
  ]
}

You are now ready to kick off a remote proxy sync. Go to Smart Proxies -> , click the "Synchonize" button and choose "Complete Sync".

Check the UI for the proxy by checkin the "Content" tab. Although the number of collections is not (currently) shown, if there was no previous content sync'd, you should now see a Library content view with a product present.

Check the pulpcore schema on the proxy:

select version, collection_id, namespace, name, repository from ansible_collectionversion;
 version |            collection_id             |   namespace    |          name           |                            repository                             
---------+--------------------------------------+----------------+-------------------------+-------------------------------------------------------------------
 4.0.6   | 548a3203-cd3f-4179-8326-9857cf6b8eb4 | testing        | ansible_testing_content | 
 4.0.4   | 548a3203-cd3f-4179-8326-9857cf6b8eb4 | testing        | ansible_testing_content |

select * from core_content;
               pulp_id                |         pulp_created          |       pulp_last_updated       |         pulp_type          | upstream_id 
--------------------------------------+-------------------------------+-------------------------------+----------------------------+-------------
 7e55c0ee-ceff-4895-a591-1496fc37dc17 | 2020-10-13 15:26:10.711397+00 | 2020-10-13 15:26:10.744853+00 | ansible.collection_version | 
 cf4244b9-9e17-4008-b95b-9dcc1d2d34da | 2020-10-13 15:26:10.716647+00 | 2020-10-13 15:26:10.775928+00 | ansible.collection_version |

select pulp_id, relative_path, artifact_id, content_id from core_contentartifact;
               pulp_id                |                relative_path                 |             artifact_id              |              content_id              
--------------------------------------+----------------------------------------------+--------------------------------------+--------------------------------------
 7264520d-dcac-44e2-b8c8-9baa4747dce6 | testing-ansible_testing_content-4.0.6.tar.gz | 44efa764-7f47-44f8-b6cd-032a0928632b | 7e55c0ee-ceff-4895-a591-1496fc37dc17
 ae5330ee-13da-44e3-8141-c178b3217c7b | testing-ansible_testing_content-4.0.4.tar.gz | 11ac4728-fd4e-4ef3-a398-52962be0b998 | cf4244b9-9e17-4008-b95b-9dcc1d2d34da

The content/artifacts should match on both the katello host and on the proxy.

Attached files from katello/pulp primary:
/etc/httpd/conf.d/05-foreman-ssl.conf
/etc/httpd/conf.d/05-foreman-ssl.d/pulp_ansible.conf

IMPORTANT for the katello master update your /etc/pulp/settings.py:

ANSIBLE_API_HOSTNAME="https://centos7-katello-devel-stable.example.com"
ANSIBLE_CONTENT_HOSTNAME="https://centos7-katello-devel-stable.example.com/pulp/content"

theforeman-bot · 2020-10-01T19:30:39Z

Issues: #27398

app/services/katello/pulp3/repository/yum.rb

sjha4 · 2020-10-20T17:59:25Z

@jjeffers :
Can you paste the versions of pulpcore and plugins on your dev box and proxy. I am seeing some issues with file proxy sync I am guessing are because of version mismatch.

jjeffers

@jjeffers :
Can you paste the versions of pulpcore and plugins on your dev box and proxy. I am seeing some issues with file proxy sync I am guessing are because of version mismatch.

Development (katello host)

{
  "versions": [
    {
      "component": "pulpcore",
      "version": "3.7.0"
    },
    {
      "component": "pulp_rpm",
      "version": "3.7.0"
    },
    {
      "component": "pulp_file",
      "version": "1.3.0"
    },
    {
      "component": "pulp_ansible",
      "version": "0.4.1"
    }
  ],

devel proxy

{
  "versions": [
    {
      "component": "pulpcore",
      "version": "3.7.1"
    },
    {
      "component": "pulp_2to3_migration",
      "version": "0.6.0.dev0"
    },
    {
      "component": "pulp_file",
      "version": "1.4.0.dev0"
    },
    {
      "component": "pulp_container",
      "version": "2.1.0"
    },
    {
      "component": "pulp_ansible",
      "version": "0.4.1"
    },
    {
      "component": "pulp_certguard",
      "version": "1.1.0.dev0"
    },
    {
      "component": "pulp_rpm",
      "version": "3.7.0"
    }
  ]

jjeffers · 2020-10-01T20:15:04Z

app/services/katello/pulp3/repository_mirror.rb

@@ -91,10 +91,11 @@ def distribution_options(path, options = {})

      def remote_options
        base_options = common_remote_options
-        base_options.merge(url: remote_feed_url)
+        base_options.merge!(url: remote_feed_url)
        if (type_specific_options = repo_service.try(:mirror_remote_options))


The bug seemed to manifest where not all options would be merged if this test failed (mirror remote options threw an exception).

I think I was hitting this..Let me test with this change..I was seeing the remote issue on master branch..

I'm not 100% convinced that it's a bug as long as the repository service sub-class provides a hash value. I would like folks to really tear down this change and if they agree it's a problem, we should change it. Otherwise, I'd rather keep the original implementation to avoid unneeded changes.

Are you able to sync file repos to the proxy?
I am hitting this:

Error: ArgumentError The input argument (attributes) must be a hash in `PulpFileClient::FileFileRemote` initialize method

http://pastebin.test.redhat.com/911972

I only tried yum. Does this happen on master?

Oh, hey I wonder if the file repository service sub-class implements the mirror_repository_options?

I started with files. Although my proxy is 3.6 pulpcore. The file plugin is up to date. If you are able to reproduce, that would be something we might want to fix for 3.6 as well?
Spinning up a 3.7 proxy to test the ansible piece.

app/services/katello/pulp3/repository/yum.rb

sjha4 · 2020-10-21T20:55:47Z

So I am seeing the error on proxy sync:
Actions::Pulp3::CapsuleContent::Sync (skipped)
https://paste.centos.org/view/9a7a0cde

I am guessing it's something to do with apache config..I can hit that URL on my master and see results..

jjeffers · 2020-10-22T09:41:59Z

So I am seeing the error on proxy sync:
Actions::Pulp3::CapsuleContent::Sync (skipped)
https://paste.centos.org/view/9a7a0cde

I am guessing it's something to do with apache config..I can hit that URL on my master and see results..

Yup, that's the same debugging route I did when I couldn't hit that URL from the proxy.

sjha4

Works well for ansible collections. Able to see content synced to proxies with changes to apache config.
Fixes file and yum capsule sync with the remote option change as well.
ACK 👍

Fixes #27938 - removing this mistake

theforeman-bot added Needs testing Not yet reviewed labels Oct 1, 2020

jjeffers changed the title ~~Fixes #27398 - adds tests for repo mirror actions~~ Fixes #27398 - enable ansible collection smart proxy content sync Oct 1, 2020

jjeffers force-pushed the 27398_enable_ansible_collection_smart_proxy_content_sync branch from 88efdba to b270a9d Compare October 1, 2020 20:16

theforeman-bot added the Waiting on contributor label Oct 15, 2020

jjeffers force-pushed the 27398_enable_ansible_collection_smart_proxy_content_sync branch from 8e30cf0 to 3a53249 Compare October 15, 2020 13:10

theforeman-bot added Waiting on contributor and removed Waiting on contributor labels Oct 15, 2020

sjha4 reviewed Oct 19, 2020

View reviewed changes

app/services/katello/pulp3/repository/yum.rb Outdated Show resolved Hide resolved

theforeman-bot added Waiting on contributor and removed Waiting on contributor labels Oct 19, 2020

jjeffers commented Oct 20, 2020

View reviewed changes

sjha4 approved these changes Oct 22, 2020

View reviewed changes

Fixes #27398 - enables remote proxy ansible collection content sync

f9d827b

Fixes #27938 - removing this mistake

jjeffers force-pushed the 27398_enable_ansible_collection_smart_proxy_content_sync branch from 2b96c46 to f9d827b Compare October 22, 2020 18:43

theforeman-bot added Waiting on contributor and removed Waiting on contributor labels Oct 22, 2020

jjeffers merged commit bc41186 into Katello:master Oct 27, 2020

jjeffers deleted the 27398_enable_ansible_collection_smart_proxy_content_sync branch October 27, 2020 12:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixes #27398 - enable ansible collection smart proxy content sync #8974

Fixes #27398 - enable ansible collection smart proxy content sync #8974

jjeffers commented Oct 1, 2020 •

edited

Loading

theforeman-bot commented Oct 1, 2020

sjha4 commented Oct 20, 2020

jjeffers left a comment •

edited

Loading

jjeffers Oct 1, 2020

sjha4 Oct 20, 2020

jjeffers Oct 20, 2020

sjha4 Oct 21, 2020

jjeffers Oct 21, 2020

jjeffers Oct 21, 2020

sjha4 Oct 21, 2020

sjha4 commented Oct 21, 2020

jjeffers commented Oct 22, 2020

sjha4 left a comment

Fixes #27398 - enable ansible collection smart proxy content sync #8974

Fixes #27398 - enable ansible collection smart proxy content sync #8974

Conversation

jjeffers commented Oct 1, 2020 • edited Loading

theforeman-bot commented Oct 1, 2020

sjha4 commented Oct 20, 2020

jjeffers left a comment • edited Loading

Choose a reason for hiding this comment

jjeffers Oct 1, 2020

Choose a reason for hiding this comment

sjha4 Oct 20, 2020

Choose a reason for hiding this comment

jjeffers Oct 20, 2020

Choose a reason for hiding this comment

sjha4 Oct 21, 2020

Choose a reason for hiding this comment

jjeffers Oct 21, 2020

Choose a reason for hiding this comment

jjeffers Oct 21, 2020

Choose a reason for hiding this comment

sjha4 Oct 21, 2020

Choose a reason for hiding this comment

sjha4 commented Oct 21, 2020

jjeffers commented Oct 22, 2020

sjha4 left a comment

Choose a reason for hiding this comment

jjeffers commented Oct 1, 2020 •

edited

Loading

jjeffers left a comment •

edited

Loading