Problem with new version of docker-py which has inpact on docker_image.present state #45292
Comments
garethgreenaway
added
Bug
|
@terminalmage Have you seen this? |
|
The only way we get to that traceback is if an unexpected exception is raised within docker-py. When docker-py catches API (or other) errors, it will raise its own custom exception class. Looking at docker-py's commit history, they did make a change involving the minimum version of requests a few weeks before releasing 2.7.0, and that's where the On Salt's end, the only thing I can think of is that when we reload the modules, we still may have a cached docker client stored in the __context__.pop('docker.client', None)Then repeat the steps you took to reproduce. Also, it's important to note whether or not the act of installing docker-py 2.7.0 resulted in upgrading |
|
Actually, that theory of mine may be negated if you didn't actually have docker-py installed when you ran those states, because we wouldn't have had the cached client in memory. |
|
I actually can't reproduce this either way, I tried locally and it "just works" whether or not I had 2.6.1 installed prior to installing 2.7.0, and it does this irrespective of the prior version of requests. I tried with requests 2.18.4 already installed, as well as by downgrading it to requests 2.6 before running the salt states. In all cases, docker-py 2.7.0 was installed (I had to pin the version in my pip.installed state to 2.7.0 since they've had a version bump to 3.x recently), and the |
terminalmage
added
the
cannot-reproduce
|
In trying to narrow down a similar issue, I've also hit this message/bug. install pip2:
pkg.purged:
- name: python-pip
cmd.script:
- name: salt://pip/files/get-pip2.py
- unless: pip --version
- require:
- pkg: install pip2
- reload_modules: True
docker-repo:
pkgrepo.managed:
- name: deb https://apt.dockerproject.org/repo {{ grains["os"]|lower }}-{{ grains["oscodename"] }} main
- humanname: {{ grains["os"] }} {{ grains["oscodename"]|capitalize }} Docker Package Repository
- keyid: 58118E89F3A912897C070ADBF76221572C52609D
- keyserver: hkp://p80.pool.sks-keyservers.net:80
- refresh_db: True
docker-engine:
pkg.installed:
- require:
- pkgrepo: docker-repo
docker:
pip.installed:
- reload_modules: True
- require:
- pkg: docker-engine
container:
docker_container.running:
- name: nginx
- image: nginx:latest
- require:
- pip: docker
- restart_policy: unless-stopped
Tested with that |
|
@terminalmage i have exactly same issue. But the problem is that it's working with 2nd round. So looks like first round did not refresh modules even if |
|
@hemebond the two log lines show that modules are being reloaded. @dawidmalina I know for a fact that module reloading works, because we have exhaustive integration tests that make sure it does. Unless I can get a reproducible test case, there's not much I can do. I tried to reproduce pretty extensively and was not able to. I'm not denying that the problem exists, but I don't have enough information on the edge case we're hitting here to be able to move forward. |
|
Wired, it started to work when i change from: to It must be somehow related dependencies. |
|
I'm no longer able to reproduce this particular error message. |
|
@terminalmage this is state which can reproduce error and we have and second try @dawidmalina is right when we are installing docker-compose instead of docker package fetch works on first iteration everything tested on salt 2018.3.2 and ubuntu 16.04 |
|
I just triggered the "Unable to perform images: cannot import name universaldetector" salt error. ID: pull down docker_image xxx pip freeze | grep dockerdocker==3.5.0 salt-call --versions-reportSalt Version: Dependency Versions: System Versions: |
|
Also hit this bug now Looking at vm without the error i down-graded two pip packages Still got the error below, restart salt-minion, the error is gone. Upgrade both back to original versions restart salt-minion, problem still gone :( Here is what the error looked like before it disappeared, tried multiple times got the same error. $ sudo pip freeze | grep "docker|request|urllib|salt" $ sudo salt-call --versions-report Dependency Versions: System Versions: |
|
One more vm with the error. Tried multiple times from salt-master, gave the same error. Fix/work around was to restart the salt-minion |
|
I am hitting the same error as well. After moving from docker-py (1.10.x) to docker (3.5.1) I constantly hit this issue when provisioning a node from scratch. The second run is always successful. Any insights on this issue? Should we use the old docker-py module instead? If yes, then maybe the docs should be updated? |
|
I've never been able to reproduce it... Are you making sure to uninstall |
|
@terminalmage Do you have an example of the states you use for testing as well as the Edit 1: running each step manually using execution modules works fine. |
|
@hemebond https://gist.github.com/terminalmage/b4e8c5998d1d6304ce1ce4bd9946aef2 I'm running this on the head of the 2018.3 branch, on Arch Linux. Version 1.10.6 of Perhaps the reason some of you are seeing this error is because you're not uninstalling |
Does it still work if you have never had |
|
Yes, it still works. I recreated the virtualenv I have been using for testing, deleted the image, and re-ran the SLS file. The result was the same. |
|
I am using kitchen ci with a bento/ubuntu-16.04 vagrant box. The vm does not contain at the beginning neither python nor any pip modules. After creating the initial vm I ran pip freeze which just failed since I haven't installed yet anything on it. Our salt setup installed before the As said already by others, right after the failure if I run a salt run again it works. EDIT: I am using a masterless workflow via kitchen-salt |
|
If you can share the Vagrantfile and steps to reproduce, I can try to do so. |
|
Hi @terminalmage, after playing a bit around I have now a combination that works:
I get the universaldetector error with
This is the vagrantfile generated by kitchen-vagrant: For the reproduction steps I will need to extract the important pieces from our setup. I will get back to it tomorrow. |
|
I still get the |
|
@martinm82 thanks! |
|
@terminalmage The sls file could be probably made smaller but at least I can use it to reproduce the issue: https://gist.github.com/martinm82/f3a31d011ed765ccc9101d0fc4614dca |
|
@martinm82 where are the files in your salt fileserver coming from? Also, salt isn't installed using the Vagrantfile you provided, so did you simply use salt-bootstrap? If so, please provide the bootstrap command you used so that I can get this as close as possible to your setup. |
|
@terminalmage I am using just salt-bootstrap. BTW, I think I might know what the problem is. I figured out that the salt modules are stored inside EDIT: Seems to be caused by the fact I am installing pip with easy_install |
|
That might be a problem. What happens if you delete pip from /usr/local/lib and install it from your apt sources? |
|
I am using now the python-pip package of ubuntu xenial which is pretty old version 8.1.1 and you won't believe it but it works. The problem is, the docker package is still installed under Before changing to system pip package I as well dumped in Salt the sys.path to see whether both |
|
The location where the custom pip packages are installed is not the problem. I am using now python-pip 8.1.1 from xenial and it still installs the packages to So the only reliable solution right now is to use a pretty old version from xenial. Any approach to update pip (with |
|
I've upgraded to 2018.3.3 on Debian Jessie and it's still an issue. A similar issue happens on Debian Stretch but it's a different module (within idna) that can't be imported. Again, restarting the minion allows it to work. |
|
closing this issue as that the Open Core Team won't fix for now since it is Python 2.7 which we will stop supporting in the Sodium release this month, and given the age of the issue. if this issue still exists with Python 3 please open a new issue. With the many changes in Salt and Docker it will help to have a updated details on the existing problem. |
|
@sagetherage It seems that you have misunderstood the nature of this issue. There is nothing Python 2-specific about it. I still don't think this is a Salt bug, though. |
Description of Issue/Question
After new release of docker-py (https://github.com/docker/docker-py/releases/tag/2.7.0), salt is not able
to install docker-py and download docker image during one iteration despite that reload_modules is set to True.
second iteration:
On older version of docker-py (2.6.1)
Salt is downloading images during first try
Setup
State which can reproduce issue
Steps to Reproduce Issue
Add repo for docker and run state
Versions Report
The text was updated successfully, but these errors were encountered: