This repository has been archived by the owner on Mar 23, 2019. It is now read-only.
Read timeout when creating a large image #147
Comments
|
same issue here! |
|
@rokroskar or @ajayaa I want to make sure we troubleshoot and fix this. Can you please provide some additional information:
Thank you for reporting this issue. We definitely want to get it solved. Having the above info will be a big help. |
|
Hello. I'm experiencing the same issue when trying to build an example project included into this repo. When I run
When trying to re-run |
|
@Wildream - That's a different issue than the one on this ticket. Presently, Docker 1.12 is not supported by Ansible Container. |
|
@chouseknecht here you go: Most of the development was done on Docker for Mac but also tried on CentOS 6 with similar results. Another oddity is that the build actually completes, the image is there, it's just not tagged properly. So if I tag it by hand to be
- hosts: jupyterhub
vars:
yum_packages: [wget, openssl, epel-release, bzip2, sudo, readline-devel]
yum_epel_packages: [nodejs, npm, python-pip]
python2_packages: [sqlalchemy, numpy, matplotlib, jupyter, notebook, ipykernel, psycopg2, pandas]
python3_packages: [sqlalchemy, tornado, jinja2, traitlets, requests, pip, ipykernel, notebook]
system_pip: /usr/bin/pip
miniconda_path: /usr/local/miniconda
conda_py2: "{{miniconda_path}}/bin/conda"
conda_py3: "{{miniconda_path}}/envs/python35/bin/conda"
conda_py2_pip: "{{miniconda_path}}/bin/pip"
conda_py3_pip: "{{miniconda_path}}/envs/python35/bin/pip"
tasks:
# basic package install
- name: install development tools
yum: name="@development-tools" state=present
- name: install basic packages
yum: name={{ item }} state=present
with_items:
- "{{yum_packages}}"
- name: install yum-epel packages
yum: name={{item}} state=present
with_items:
- "{{yum_epel_packages}}"
# miniconda
- name: download miniconda installer
get_url:
url=https://repo.continuum.io/miniconda/Miniconda2-4.1.11-Linux-x86_64.sh
dest=/tmp/miniconda.sh
mode=0755
checksum="md5:b2af3b9ff39c4a4a812f50cecbafcda6"
- name: install miniconda
shell: "/tmp/miniconda.sh -b -p {{miniconda_path}} creates={{miniconda_path}} executable=/bin/bash"
- name: conda install python2 packages
conda: name={{item}} state=latest executable={{conda_py2}}
with_items: "{{python2_packages}}"
- name: make a python 3 environment
shell: "{{conda_py2}} create --yes -q -n python35 python=3.5 creates={{miniconda_path}}/envs/python35"
- name: install python3 packages
conda: name={{item}} executable={{conda_py3}} state=latest
with_items: "{{python3_packages}}"
- name: remove miniconda installer
file: path=/tmp/miniconda.sh state=absent
# set up jupyterhub user to run jupyterhub instead of root
- name: create group shadow
group: name=shadow state=present
- name: change permissions on the shadow file
file: path=/etc/shadow group=shadow mode=g=r
- name: create jupyterhub user
user: name=jupyterhub groups=shadow append=yes createhome=no
- name: set up sudoers
template: dest=/etc/sudoers src=./sudoers owner=root group=root
- name: install sudospawner
pip: executable={{conda_py3_pip}} name=https://github.com/jupyterhub/sudospawner/archive/master.zip
# jupyterhub
- name: install http proxy
npm: name=configurable-http-proxy global=yes
- name: install jupyterhub
pip: name=jupyterhub executable={{conda_py3_pip}}
- name: make jupyterhub directory
file: path=/usr/local/jupyterhub state=directory owner=jupyterhub group=jupyterhub
- name: create jupyterhub configuration
template: dest=/usr/local/jupyterhub/jupyterhub_config.py src=./jupyterhub_config.py owner=jupyterhub group=jupyterhub
- name: make ssl certs
shell: openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout /usr/local/jupyterhub/jhub.key -out /usr/local/jupyterhub/jhub.cert -batch
args:
creates: /usr/local/jupyterhub/jhub.key
# register python kernels
- name: register py2 kernel
shell: "{{miniconda_path}}/bin/python -m ipykernel install"
args:
creates: /usr/local/share/jupyter/kernels/python2
- name: register py3 kernel
shell: "{{miniconda_path}}/envs/python35/bin/python -m ipykernel install"
args:
creates: /usr/local/share/jupyter/kernels/python3
# create users and make links to /notebooks
- name: copy notebooks to container volume
copy: dest=/ src=../files/notebooks mode=u=rwX,g=rX,o=rX
- name: create user test
user:
name: test
groups: jupyterhub
append: yes
shell: /bin/bash
- name: create user notebook directories
file: dest=/notebooks/test state=directory owner=test group=jupyterhub
- name: create links to notebook directory
file: src=/notebooks dest=/home/{{item}}/notebooks state=link
with_items:
- test
# install stata
- name: pip install pexpect needed to install stata
pip: name=pexpect state=latest executable=/usr/bin/pip
- name: make stata directory
file: path=/usr/local/stata14 state=directory
- name: unarchive stata
unarchive: src=files/Stata14Linux64.tar.gz dest=/usr/local/stata14 creates=/usr/local/stata14/install
- name: fix stata install file
lineinfile: dest=/usr/local/stata14/install line="#!/bin/sh" insertbefore=BOF state=present
- name: install stata
expect:
chdir: /usr/local/stata14
command: /usr/local/stata14/install
responses:
"y/n": "y\n"
creates: /usr/local/stata14/stinit
timeout: 60
- name: pip install ipystata
pip: name=ipystata executable={{conda_py2_pip}}
- name: install libpng12
yum: name=libpng12 state=present
# install R
- name: install R
yum: name=R.x86_64 state=present
- name: pip install rpy2
pip: name=rpy2 executable={{conda_py2_pip}}
version: "1"
services:
jupyterhub:
image: centos
ports:
- "9000:8000"
command: "jupyterhub"
working_dir: /usr/local/jupyterhub
user: jupyterhub
environment:
PATH: /usr/local/miniconda/envs/python35/bin:$PATH"
volumes:
- /notebooks
registries: {}
|
|
@rokroskar , You can increase the docker-compose read timeout. I was having the same problem, worked for me. |
|
@procrypt Increasing the timeouts doesn't work here. System is Ubuntu 16.04.1, Docker Engine's in use. The vivado_install role downloads a 9GB tar.gz of Vivado's installer, untars it, installs it, then deletes the tar and decompressed dir. The resulting image is 15GB.
|
|
docker-compose sets timeout via environment variable on creating but ansible-container doesn't so I tried setting timeout to 3000 for now and could commit a image successfully. |
|
This started happening to me today, with a container over 2GB, after I changed the storage driver from devicemapper to overlay2, on Arch Linux. The failure happens during the commit. I bumped up EDIT: Yes, hard-coding the timeout as rinatz demonstrated allowed me to move past the issue for now. (Thanks rinatz!) |
|
Tracking upgrade to Docker 1.12 on #314 |
Merged
|
It looks like this issue was originally resolved by #318, but I'm building a 1.8GB image in ansible-container 0.9.1 and getting the same behavior as this user originally reported - but neither exporting COMPOSE_HTTP_TIMEOUT=3000 to my outer "build" environment nor adding it to the If I add the The reason for the regression seems to be a switch from directly instantiating
...container builds as expected...
|
This was referenced Jul 22, 2017
chouseknecht
pushed a commit
that referenced
this issue
Jul 22, 2017
|
I encounter the same bug even though I use version which contains fixes mentioned in this issue. The minimal example is an Ansible role with single task: With such Ansible role, the last step which commits role as a layer (seen it in the traceback), repeatedly end up with error message: I tried setting environment variable e.g. However, when I set the env. variable to something ridiculously small e.g. I looked into the source codes in my Python virtualenv and I am sure that this code reads timeout value from the environment (i.e. pull requests 660, 661). It seems that for some reason, the last action of creating the layer makes a reset of timeout parameter. Here is the content of The role Here is the result of |
|
Actually, it is even simpler:
With ansible-container 0.9.2, regardless of any env. vars., the above example always fails with |
|
I am experimenting the exact same issue with large containers with ansible-container 0.9.3rc0 and no environment variable could help. After Please re-open this issue. |
|
@Chostakovitch, I have found a solution to this problem. The environment must be set on the conductor image, and not on your machine! Here are the steps to solve your problem. Prepare custom conductor image:git clone https://github.com/ansible/ansible-container.git
echo 'ENV DOCKER_CLIENT_TIMEOUT=600' >> ansible-container/container/docker/templates/conductor-local-dockerfile.j2Fix for pip v10sed -i 's/from pip.req import parse_requirements/try:\n from pip._internal.req import parse_requirements\nexcept ImportError:\n from pip.req import parse_requirements/g' ansible-container/setup.pyCreate virtualenvvirtualenv venv
source venv/bin/activateInstall ansible-container from your own custom directory (note the
|
|
@tzok Thanks a lot, this made the trick for the moment. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I'm creating a large image (~3Gb) and the
ansible-container buildfails at the very end withI tried increasing the timeout using environment variables, as described e.g. here but this doesn't seem to have any effect.
Interestingly, if I do a partial build on the first pass by leaving out some components and bringing the image size below ~ 2Gb, it works. I can then add the other components back in building on top of the previous image and this build finishes fine (now composed of the two layers instead of one).
versions:
System is mac os x 10.10.5.
The text was updated successfully, but these errors were encountered: