Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apt_repository module does not work behind proxy on ubuntu #42534

Closed
hangsu-ma opened this issue Jul 9, 2018 · 9 comments
Closed

apt_repository module does not work behind proxy on ubuntu #42534

hangsu-ma opened this issue Jul 9, 2018 · 9 comments
Labels
affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. has_pr This issue has an associated PR. module This issue/PR relates to a module. packaging Packaging category support:core This issue/PR relates to code supported by the Ansible Engineering Team.

Comments

@hangsu-ma
Copy link

SUMMARY

apt_repository is calling apt-key when adding new repo source. apt-key does not respect http_proxy env var nor Acquire::http::Proxy specified in apt conf files. However it does support proxy setting passed in from --keyserver-options parameter. Current apt_repository is missing logic to handle keyserver-options.
Therefore apt_repository will not be able to add new repo behind proxy on ubuntu

ISSUE TYPE
  • Bug Report
COMPONENT NAME

apt_repository

ANSIBLE VERSION
all
CONFIGURATION
OS / ENVIRONMENT

target env: ubuntu

STEPS TO REPRODUCE
- apt_repository:
    repo: 'ppa:git-core/ppa'
    state: absent
    validate_certs: false
    update_cache: yes
  environment:
    http_proxy: http://proxy.example.com:8080
  become: yes
EXPECTED RESULTS

repo been added

ACTUAL RESULTS
failing on apt-key command
@ansibot
Copy link
Contributor

ansibot commented Jul 9, 2018

Files identified in the description:

If these files are inaccurate, please update the component name section of the description or use the !component bot command.

click here for bot help

@ansibot
Copy link
Contributor

ansibot commented Jul 9, 2018

cc @sashka
click here for bot help

@ansibot ansibot added affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. module This issue/PR relates to a module. needs_triage Needs a first human triage before being processed. support:core This issue/PR relates to code supported by the Ansible Engineering Team. labels Jul 9, 2018
@hangsu-ma hangsu-ma mentioned this issue Jul 9, 2018
Closed
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Jul 9, 2018
@hangsu-ma
support add repo behind proxy on ubuntu(ansible#42534)
4e8bf6a 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect http_proxy environment variable.
As a result, one would thought adding http_porxy env var using environment
will be sufficient, which is never the case and apt_repository will always fail
to add new repo behind proxy on ubuntu.

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix read http_proxy environment variable and pass it on to apt-key using --keyserver-options.
@webknjaz webknjaz removed the needs_triage Needs a first human triage before being processed. label Jul 10, 2018
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Jul 31, 2018
@hangsu-ma
handle no_proxy on ubuntu(ansible#42534)
524f66e 
apt_repository module calls apt-key while adding new repository.
apt-key will use default ubuntu key server(keyserver.ubuntu.com) and apt-key does not support any parameter to pass in no_proxy settings.
In a very unlikely situation, where default ubuntu key server is bypassing proxy and can't be accessed through proxy, and the playbook has key server in no_proxy.
By only taking in account of http_proxy, apt_key would been invoked with proxy even the key server is in no_proxy settings.
This fix addresses this, it will check if default ubuntu key server is in no_proxy list before passing proxy settings to apt-key.
NOTE: default ubuntu key server used in this module is keyserver.ubuntu.com, no DNS look up is carried out here, all IP/subnet in no_proxy will be ignored.
This means even key server's IP is in no_proxy list, proxy will still be set for apt-key.
NOTE: Due to the lack of definition for no_proxy, this fix supports:
1. no_proxy is a comma separated list of one or more items below
2. full domain name or hostname
3. domain name or hostname containing * wildcard
4. domain name or hostname with leading dot (.)
@OurFriendIrony
Copy link

Has this been resolved yet, or a clean workaround defined? I'm having the same issue

@hangsu-ma
Copy link
Author

Has this been resolved yet, or a clean workaround defined? I'm having the same issue

@OurFriendIrony
The PR is in progress. If you want to apply the workaround, you can create a folder called library in same folder where your playbook is. Then copy the fixed version from #42536 in the folder. Ansible will then automatically choose the version in library folder over the default one at run time, see more details here: https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html#adding-a-module-locally

@ansibot ansibot added the packaging Packaging category label Feb 17, 2019
This was referenced Mar 22, 2019
Merged
Closed
@Abam
Copy link

Abam commented May 29, 2019

Having Same issue on Ubuntu 18.04.2, apt_repository implementation is using apt-key without the proxy env.

@achamayou achamayou mentioned this issue Jun 5, 2019
Closed
@ansibot ansibot added the has_pr This issue has an associated PR. label Jul 24, 2019
@Abam
Copy link

Abam commented Aug 22, 2019

Any plan to solve this annoying issue ?

@colinscz colinscz mentioned this issue Nov 8, 2019
Merged
@freeeflyer
Copy link

freeeflyer commented Mar 4, 2020 •
edited

@hangsu-ma

Is the PR still in progress ?

Has this been resolved yet, or a clean workaround defined? I'm having the same issue

@OurFriendIrony
The PR is in progress. If you want to apply the workaround, you can create a folder called library in same folder where your playbook is. Then copy the fixed version from #42536 in the folder. Ansible will then automatically choose the version in library folder over the default one at run time, see more details here: https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html#adding-a-module-locally

hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Apr 9, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
d857fa9 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Apr 10, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
29536b0 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Apr 10, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
5dec9bd 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue Apr 10, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
d5149e3 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
@g2flyer g2flyer mentioned this issue May 12, 2020
Merged
@ansibot
Copy link
Contributor

ansibot commented May 16, 2020

Files identified in the description:

If these files are incorrect, please update the component name section of the description or use the !component bot command.

click here for bot help

hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue May 16, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
423ebb9 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
hangsu-ma added a commit to hangsu-ma/ansible that referenced this issue May 20, 2020
@hangsu-ma
adding proxy support for apt_repository (ansible#42534)
37635cf 
Similar reason behind ansible#42443
apt_repository module calls apt-key to add new repo source on ubuntu.
apt-key does not respect Acquire::http::Proxy specified in apt conf files, nor http_proxy environment variable.
More discussion about these behaviours can be found here: https://bugs.launchpad.net/ubuntu/+source/software-properties/+bug/1433761

keyserver-options are used to pass in proxy settings for apt-key, example:
sudo apt-key adv --keyserver-options http-proxy=http://username:password@proxy.example.com:8080 --keyserver keyserver.ubuntu.com --recv-keys GPG_KEY

This fix parse http_proxy and no_proxy environment variables and pass on proxy to apt-key using --keyserver-options if ubuntu key server is not in no_proxy list.
kai2nenobu added a commit to kai2nenobu/dotfiles that referenced this issue Feb 8, 2021
@kai2nenobu
GPG鍵を追加してからkelleyk/emacsリポジトリを追加する
d2f3bcd 
もとは単独のapt_repositoryモジュールで追加しようとしていた。
正しく動けばそれでも大丈夫なはずなんだけど、Windows10 1809、Ubuntu
18.04だとうまく動かなかった。
WSLで dirmngr が動かないとかansibleのapt_key、apt_repositoryモジュールが
プロキシ環境変数を参照しないとかが原因っぽい。
apt_repositoryモジュール単独じゃなくて

    - name: Add a gpg key for emacs ppa repository
      apt_key:
        keyserver: keyserver.ubuntu.com
        id: EAAFC9CD
        state: present
    - name: Add emacs ppa repository
      apt_repository:
        repo: ppa:kelleyk/emacs
        state: present

みたいな感じでapt_keyとapt_repositoryの組み合わせもうまくいかなかった。
"repo: ppa:kelleyk/emacs" という書き方だと、GPG鍵を改めて取得しにいって、
失敗している感じだった。
Windows10 1809でansibleのバグ修正がリリースされればapt_repositoryモジュール
単独でうまくいくかもしれない。

バグのissue

- ttps://github.com/ansible/ansible/issues/42534
- ttps://github.com/microsoft/WSL/issues/3286#issuecomment-402594992

PPAリポジトリをapt-add-repositoryを使わずに追加する話。

- https://hnakamur.github.io/blog/2017/09/02/add-ppa-to-apt-line-without-add-apt-repository/
@relrod
Copy link
Member

relrod commented Mar 26, 2021

This module now uses fetch_url which now supports proxies, so this should be fixed in current versions.

@relrod relrod closed this as completed Mar 26, 2021
@ansible ansible locked and limited conversation to collaborators Apr 23, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
affects_2.7 This issue/PR affects Ansible v2.7 bug This issue/PR relates to a bug. has_pr This issue has an associated PR. module This issue/PR relates to a module. packaging Packaging category support:core This issue/PR relates to code supported by the Ansible Engineering Team.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

support add repo behind proxy on ubuntu(#42534) hangsu-ma/ansible
7 participants
@relrod @webknjaz @hangsu-ma @Abam @ansibot @OurFriendIrony @freeeflyer