HTTP proxy specified in /etc/rhn/rhn.conf not taken by DEB spacewalk-repo-sync module

[phibid]

Uyuni installed version: 2020.03

All the RPM repositories are being correctly synced though the Web UI, which is not the case of the DEB repos. The synchronisation always ends up with:

2020/04/06 20:29:44 +02:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'ubuntu-1804-amd64-main-uyuni', '--type', 'deb', '--non-interactive'] 2020/04/06 20:29:44 +02:00 Sync of channel started. 2020/04/06 20:34:06 +02:00 Unhandled error occurred: HTTPConnectionPool(host='archive.ubuntu.com', port=80): Max retries exceeded with url: /ubuntu/dists/bionic/main/binary-amd64/Packages.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fba66f62358>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

The execution of spacewalk-repo-sync directly through CLI is correctly working, so it seems that the http_proxy parameter specified in /etc/rhn/rhn.conf is not taken into account:

server.satellite.http_proxy = proxy.zig.zag:8080

The issue #1095 regarding this parameter and its implementation in the DEB module were solved, so not sure if there is another issue here.

[rpasche]

Can confirm this error with latest uyuni server.

[rpasche]

Just manually tried this command and get following error (all updates have been applied)

/usr/bin/spacewalk-repo-sync -vvv --channel buster_security_main --type deb
11:52:43 ======================================
11:52:43 | Channel: buster_security_main
11:52:43 ======================================
11:52:43 Sync of channel started.
11:52:44 Plugin error: Package index checksum failure
Traceback (most recent call last):
  File "/usr/bin/spacewalk-repo-sync", line 286, in <module>
    sys.exit(abs(main() or 0))
  File "/usr/bin/spacewalk-repo-sync", line 269, in main
    elapsed_time, channel_ret_code = sync.sync()
  File "/usr/lib/python3.6/site-packages/spacewalk/satellite_tools/reposync.py", line 667, in sync
    if CFG.AUTO_GENERATE_BOOTSTRAP_REPO:
  File "/usr/lib/python3.6/site-packages/spacewalk/common/rhnConfig.py", line 237, in __getattr__
    raise AttributeError(key)
AttributeError: AUTO_GENERATE_BOOTSTRAP_REPO

[mcalmer]

Well, the proxy issue is something different. We look into it.

The AttributeError: AUTO_GENERATE_BOOTSTRAP_REPO was already reported and is fixed in git. See #2094

[rpasche]

I just commented the verify function. Then, I can sync again (at least only checked from CLI). From WebUI, test needs to be done. Also note the TODO line. So I wonder, if this is only "accidently" activated, in case it is really not yet fully implemented.

# TODO: Signature verification is not yet implemented here.
         #if not repo.DpkgRepo(self.url).verify_packages_index():
         #    raise repo.GeneralRepoException("Package index checksum failure")

[rpasche]

But I also needed to set an

auto_generate_bootstrap_repo =

setting within /etc/rhn/rhn.conf, even with the fix in #2094.

Update to above: Without above setting, the bootstrap repo gets recreated. So if I want to deactivate auto-regeneration, empty value needs to be added to rhn.conf

Confirmed work of sync also triggered from WebUI:

2020/04/07 12:40:05 +02:00 Command: ['/usr/bin/spacewalk-repo-sync', '--channel', 'buster_main_main', '--type', 'deb', '--non-interactive']
2020/04/07 12:40:05 +02:00 Sync of channel started.
2020/04/07 12:40:09 +02:00 Repo URL: http://ftp.de.debian.org/debian/dists/buster/main/binary-amd64/
2020/04/07 12:40:09 +02:00     Packages in repo:             56885
2020/04/07 12:43:10 +02:00     No new packages to sync.
2020/04/07 12:43:11 +02:00
2020/04/07 12:43:11 +02:00   Patches in repo: 0.
2020/04/07 12:43:11 +02:00 Sync completed.

[mcalmer]

In my tests it was working. The old spacewalk config parsing uses namespaces. repo-sync use server.susemanager .

So it take values from rhn.conf without prefix and with prefix server.susemanager.
From config-defaults it uses rhn_server_susemanager.conf as default file.

The problem was, that in deb_src.py the namespace was switched to read proxy values.
The fix switch it back and this should make it possible to read the variable also from the default template.
Please check if rhn_server_susemanager.conf contains that key.

[phibid]

Nothing related to proxy in /usr/share/rhn/config-defaults/rhn_server_susemanager.conf file:

uyuni:~ # egrep -v '^$|#' /usr/share/rhn/config-defaults/rhn_server_susemanager.conf
scc_url = https://scc.suse.com
forward_registration = 1
fromdir =
mirror =
virtpoller.cache_file = /var/cache/virt_state.cache
virtpoller.expire_time = 21600
virtpoller.interval = 300
temp_token_lifetime = 60
token_lifetime = 525600
token_refresh_auto_deploy = true
bootstrap_repo_flush = 1
bootstrap_repo_datamodule = mgr_bootstrap_data
auto_generate_bootstrap_repo = 1

I have tried to add http_proxy and server.satellite.http_proxy (uyuni restarted after each modification) but no luck.

Thanks !

[mcalmer]

@phibid the proxy problem is different. We were able to see it here as well. So we look into it. But may take a bit. There was some new code added which does not take the proxy into account.

[rpasche]

Found the error. In Debian Release file, there is no more sha1 checksum present. So the check for this checksum is failing and thus returning an error. In file /usr/lib/python3.6/site-packages/spacewalk/common/repo.py changing line 224 to

for algorithm in ["md5", "sha256"]:

(remove sha1) fixes the problem.

[phibid]

@rpasche I confirm your issue and the fix for Debian repos. Thanks for this. Perhaps that you should create an issue if not already done ?

@mcalmer Ah ok, thought you were talking about mine. Anyway, adding the proxy settings directly at the beginning of /usr/bin/spacewalk-repo-sync just after "import os" is a working workaround in my case:

proxy = 'http://my_proxy_ip:my_proxy_port'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy

[phibid]

@rpasche Had to remove "md5" too in /usr/lib/python3.6/site-packages/spacewalk/common/repo.py, as debian-10-amd64-main-updates repo sync were failing otherwise.

[rpasche]

@phibid really? Have to check this...

[rpasche]

@phibid You're right. The buster-updates part does only contain SHA256.

[rpasche]

@mcalmer
I can confirm that this error still exists in 2020.04. Trying to sync a debian repo currently seems to freeze.

uyuni:~ # ss -a4n | grep SYN
tcp    SYN-SENT   0      1      10.127.101.92:43624              141.76.2.4:80
uyuni:~ # host ftp.de.debian.org
ftp.de.debian.org has address 141.76.2.4
uyuni:~ #

uyuni:~ # cat /etc/uyuni-release
Uyuni release 2020.04
uyuni:~ # grep satellite /etc/rhn/rhn.conf
server.satellite.rhn_parent = satellite.rhn.redhat.com
server.satellite.http_proxy = proxy:port
server.satellite.http_proxy_username =
server.satellite.http_proxy_password =
server.satellite.no_proxy =
server.satellite.ca_chain = /usr/share/rhn/RHNS-CA-CERT
web.satellite = 1
web.satellite_install =
uyuni:~ #

What seems interesting....this seems only to happen for repos (and channels), that are not handled via the "products". All SLES repos (handled by products) are using the proxy successfully. successfully.

But I currently can not confirm this with some Ubuntu repos (that could be enabled).

[rpasche]

@mcalmer

It looks the error is simply, that no proxies are given to the requests.get() call within get_pkg_index_raw in /usr/lib/python3.6/site-packages/spacewalk/common/repo.py

    def get_pkg_index_raw(self) -> typing.Tuple[str, bytes]:
        """
        Get Packages.gz or Packages.xz or Packages content, raw.

        :return: bytes of the content
        """
        if self._pkg_index[0] == "":
            for cnt_fname in [DpkgRepo.PKG_GZ, DpkgRepo.PKG_XZ, DpkgRepo.PKG_RW]:
                packages_url = self.append_index_file(cnt_fname)
                resp = requests.get(packages_url)
                if resp.status_code == http.HTTPStatus.OK:
                    self._pkg_index = cnt_fname, resp.content
                    break
                resp.close()

The proxy should be defined an passed to requests.get as within _download function within /usr/lib/python3.6/site-packages/spacewalk/satellite_tools/repo_plugins/deb_src.py

        for _ in range(0, RETRIES):
            try:
                proxies=""
                if self.proxy:
                    (scheme, netloc, path, query, fragid) = urlparse.urlsplit(self.proxy)
                    proxies = {
                        'http': 'http://' + netloc,
                        'https': 'http://' + netloc
                    }
                    if self.proxy_username and self.proxy_password:
                        proxies = {
                            'http': 'http://' + self.proxy_username + ":" + self.proxy_password + "@" + netloc,
                            'https': 'http://' + self.proxy_username + ":" + self.proxy_password + "@" + netloc,
                        }
                data = requests.get(url, proxies=proxies, cert=(self.sslclientcert, self.sslclientkey),
                                    verify=self.sslcacert)
                if not data.ok:

[rpasche]

It also looks that the get_release_index would also have to be updated with proxy settings

    def get_release_index(self) -> typing.Dict[str, "DpkgRepo.ReleaseEntry"]:
        """
        Find and return contents of Release file.

        :raises GeneralRepoException if the Release file cannot be found.
        :return: string
        """
        resp = requests.get(self._get_parent_url(self._url, 2, "Release"))
        try:
            self._flat = resp.status_code in [http.HTTPStatus.NOT_FOUND, http.HTTPStatus.FORBIDDEN]
            self._flat_checked = 1

[admd]

Hi @rpasche

Thank you for all these findings. We will have a look. Besides that, PRs are always welcome from the community.

[phibid]

Not sure why this issue is closed, I have the same issue on Uyuni 2020.05:

1 - On file /usr/lib/python3.6/site-packages/spacewalk/common/repo.py, line 224 needs to be modified as follow :

for algorithm in ["sha256"]:

To allow correct synchronization of Debian/Ubuntu repos

2 - Proxy settings are not taken for .DEB repos, I currently have to add this in file /usr/bin/spacewalk-repo-sync as a workaround:

proxy = 'http://my_proxy_ip:my_proxy_port'
os.environ['http_proxy'] = proxy
os.environ['HTTP_PROXY'] = proxy
os.environ['https_proxy'] = proxy
os.environ['HTTPS_PROXY'] = proxy

[phibid]

Ah ok, I guess that the PRs will be added in the next Uyuni maintenance release.

Not better to let this ticket opened until we validate that this is solved in the next release ?

[brejoc]

@phibid I've split the issues. The first part about not using the proxy setting was merged to master and so this got closed. The second issue is this one. There is also a PR with a draft for that: #2290

Right! Everything that gets merged now will be in the next release. As soon as a change is tested and reviewed we close the issue. Otherwise we'd lose overview fast. But you can check the changelogs to see if there is something pending.

[mcalmer]

Ah ok, I guess that the PRs will be added in the next Uyuni maintenance release.

Yes, will be part of 2020.06

Not better to let this ticket opened until we validate that this is solved in the next release ?

No, we will alwas forget and the result is just a lot of open issues which are solved since ages.
When you test it and find out that it is not working, check if the reason is the same and either reopen the issue or create a new one for a different reason.

[phibid]

Understood, thank you both !