This repository has been archived by the owner. It is now read-only.

Registry fails on startup : `OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'` #892

Open
ankushagarwal opened this Issue Jan 9, 2015 · 39 comments

Comments

Projects
None yet
@ankushagarwal

ankushagarwal commented Jan 9, 2015

I have seen similar issues related to registry here, but this one seems different. I ran a docker run -d --name registry -p 80:5000 registry on my Mac in boot2docker.

$ docker version                                                                                                                                                                                                          
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8
$ docker logs registry                                                                                                             
[2015-01-09 21:22:06 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-01-09 21:22:06 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-01-09 21:22:06 +0000] [1] [INFO] Using worker: gevent
[2015-01-09 21:22:06 +0000] [14] [INFO] Booting worker with pid: 14
[2015-01-09 21:22:06 +0000] [15] [INFO] Booting worker with pid: 15
[2015-01-09 21:22:06 +0000] [16] [INFO] Booting worker with pid: 16
[2015-01-09 21:22:06 +0000] [21] [INFO] Booting worker with pid: 21
09/Jan/2015:21:22:06 +0000 WARNING: Cache storage disabled!
09/Jan/2015:21:22:06 +0000 WARNING: Cache storage disabled!
09/Jan/2015:21:22:06 +0000 WARNING: Cache storage disabled!
09/Jan/2015:21:22:06 +0000 WARNING: LRU cache disabled!
09/Jan/2015:21:22:06 +0000 WARNING: LRU cache disabled!
09/Jan/2015:21:22:06 +0000 WARNING: LRU cache disabled!
09/Jan/2015:21:22:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
09/Jan/2015:21:22:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
09/Jan/2015:21:22:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
09/Jan/2015:21:22:06 +0000 WARNING: Cache storage disabled!
09/Jan/2015:21:22:06 +0000 WARNING: LRU cache disabled!
09/Jan/2015:21:22:06 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
09/Jan/2015:21:22:06 +0000 WARNING: Another process is creating the search database
09/Jan/2015:21:22:06 +0000 WARNING: DB is disconnected. Reconnect to it.
09/Jan/2015:21:22:06 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-01-09 21:22:06 +0000] [14] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in <module>
    from .search import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in <module>
    INDEX = index.load(cfg.search_backend.lower())
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/__init__.py", line 82, in load
    return db.SQLAlchemyIndex()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in __init__
    self._setup_database()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
    os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in <module>
    from .search import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in <module>
    INDEX = index.load(cfg.search_backend.lower())
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/__init__.py", line 82, in load
    return db.SQLAlchemyIndex()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in __init__
    self._setup_database()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
    os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-01-09 21:22:06 +0000] [21] [ERROR] Exception in worker process:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in <module>
    from .search import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in <module>
    INDEX = index.load(cfg.search_backend.lower())
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/__init__.py", line 82, in load
    return db.SQLAlchemyIndex()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in __init__
    self._setup_database()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
    os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
    super(GeventWorker, self).init_process()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
    __import__(module)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in <module>
    from .search import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in <module>
    INDEX = index.load(cfg.search_backend.lower())
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/__init__.py", line 82, in load
    return db.SQLAlchemyIndex()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in __init__
    self._setup_database()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
    os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-01-09 21:22:06 +0000] [14] [INFO] Worker exiting (pid: 14)
[2015-01-09 21:22:06 +0000] [21] [INFO] Worker exiting (pid: 21)
[2015-01-09 21:22:07 +0000] [16] [INFO] Worker exiting (pid: 16)
[2015-01-09 21:22:07 +0000] [15] [INFO] Worker exiting (pid: 15)
Traceback (most recent call last):
  File "/usr/local/bin/gunicorn", line 11, in <module>
    sys.exit(run())
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run
    WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 185, in run
    super(Application, self).run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 71, in run
    Arbiter(self).run()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 196, in run
    self.halt(reason=inst.reason, exit_status=inst.exit_status)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 292, in halt
    self.stop()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 343, in stop
    time.sleep(0.1)
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 209, in handle_chld
    self.reap_workers()
  File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in reap_workers
    raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
@ankushagarwal

This comment has been minimized.

Show comment
Hide comment
@ankushagarwal

ankushagarwal Jan 9, 2015

I have seen similar issues(#696, #829, #796, etc.) related to registry startup in Issues, but I was not sure if this was exactly the same as none of them throw the error [Errno 2] No such file or directory: './registry._setup_database.lock'

ankushagarwal commented Jan 9, 2015

I have seen similar issues(#696, #829, #796, etc.) related to registry startup in Issues, but I was not sure if this was exactly the same as none of them throw the error [Errno 2] No such file or directory: './registry._setup_database.lock'

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jan 9, 2015

Member

This is different.
I'll look into it.

Member

dmp42 commented Jan 9, 2015

This is different.
I'll look into it.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jan 9, 2015

Member

@ankushagarwal Does this happen consistently?

Member

dmp42 commented Jan 9, 2015

@ankushagarwal Does this happen consistently?

@ankushagarwal

This comment has been minimized.

Show comment
Hide comment
@ankushagarwal

ankushagarwal Jan 10, 2015

Tried reproducing it without any success.

ankushagarwal commented Jan 10, 2015

Tried reproducing it without any success.

@ankushagarwal

This comment has been minimized.

Show comment
Hide comment
@ankushagarwal

ankushagarwal Jan 10, 2015

It happens sometimes. I can't reproduce it deterministically.

ankushagarwal commented Jan 10, 2015

It happens sometimes. I can't reproduce it deterministically.

@ankushagarwal

This comment has been minimized.

Show comment
Hide comment
@ankushagarwal

ankushagarwal Jan 10, 2015

I can reproduce it deterministically.

Whenever I restart boot2docker and do a docker start registry. It happens and the registry container crashes. I then do a docker start registry for the second time and it works fine.

This is my registry config. The only change I have made from the default is on line 214

This is my b2d version :

$ boot2docker version
Boot2Docker-cli version: v1.4.1
Git commit: 43241cb

ankushagarwal commented Jan 10, 2015

I can reproduce it deterministically.

Whenever I restart boot2docker and do a docker start registry. It happens and the registry container crashes. I then do a docker start registry for the second time and it works fine.

This is my registry config. The only change I have made from the default is on line 214

This is my b2d version :

$ boot2docker version
Boot2Docker-cli version: v1.4.1
Git commit: 43241cb
@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jan 12, 2015

Member

Thanks.
Weird that it would crash there then. I'll have another look.

Member

dmp42 commented Jan 12, 2015

Thanks.
Weird that it would crash there then. I'll have another look.

@dbason

This comment has been minimized.

Show comment
Hide comment
@dbason

dbason Jan 12, 2015

Got the same thing 2 times in a row then it worked the 3rd time. @dmp42 do you want me to provide anymore information about our environment or have you got enough to go on here?

dbason commented Jan 12, 2015

Got the same thing 2 times in a row then it worked the 3rd time. @dmp42 do you want me to provide anymore information about our environment or have you got enough to go on here?

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jan 12, 2015

Member

The code just looks racy from the get go...

Member

dmp42 commented Jan 12, 2015

The code just looks racy from the get go...

@pavelz

This comment has been minimized.

Show comment
Hide comment
@pavelz

pavelz Jan 14, 2015

does not work for me either newest or 0.8.1 to 0.9.1

Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.2
Git commit (client): 54d900a
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.2
Git commit (server): 54d900a

pavelz commented Jan 14, 2015

does not work for me either newest or 0.8.1 to 0.9.1

Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.2
Git commit (client): 54d900a
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.2
Git commit (server): 54d900a

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jan 14, 2015

Member

@pavelz can you elaborate on what doesn't work?
This specific code path is not in 0.8.1 - so, you are likely having a different issue here.

Member

dmp42 commented Jan 14, 2015

@pavelz can you elaborate on what doesn't work?
This specific code path is not in 0.8.1 - so, you are likely having a different issue here.

@larrycai

This comment has been minimized.

Show comment
Hide comment
@larrycai

larrycai Jan 15, 2015

+1 for boot2docker/Windows for this error, and it disappear in 2nd time.
registry:0.9.1
boot2docker 1.4.1

larrycai commented Jan 15, 2015

+1 for boot2docker/Windows for this error, and it disappear in 2nd time.
registry:0.9.1
boot2docker 1.4.1

@rseymour

This comment has been minimized.

Show comment
Hide comment
@rseymour

rseymour Jan 16, 2015

Happens to somewhat randomly to me on docker:1.4.1.
btrfs on fedora 21.

Should note that adding

-e GUNICORN_OPTS=[--preload]

from issue #796 comments worked fine

rseymour commented Jan 16, 2015

Happens to somewhat randomly to me on docker:1.4.1.
btrfs on fedora 21.

Should note that adding

-e GUNICORN_OPTS=[--preload]

from issue #796 comments worked fine

@jasonmartens

This comment has been minimized.

Show comment
Hide comment
@jasonmartens

jasonmartens Jan 30, 2015

I am experiencing this problem running the registry on CoreOS. The first time I run the registry and it has to pull from the registry image from the docker public repository, it always seems to fail. After the images have been pulled, then it seems to start just fine.

jasonmartens commented Jan 30, 2015

I am experiencing this problem running the registry on CoreOS. The first time I run the registry and it has to pull from the registry image from the docker public repository, it always seems to fail. After the images have been pulled, then it seems to start just fine.

@weihanwang

This comment has been minimized.

Show comment
Hide comment
@weihanwang

weihanwang Feb 1, 2015

I reproduced this issue fairly consistently (~1 out of 4) on both boot2docker and a ubuntu 14.04 host. Both run docker 1.4.1 and registry 0.9.1. The same problem happened in earlier versions of docker and registry as well.

I used the default configurations of both docker and registry.

The script that triggered this issue creates a registry container:

$ docker create -P --name registry-container registry

And then repeats these steps over and over again:

  1. docker start registry-container
  2. on localhost, push ~30 containers to registry
  3. on a remote host, pull all those containers from registry
  4. docker stop registry-container
  5. go to step 1

Let me know if there's more info I can provide.

weihanwang commented Feb 1, 2015

I reproduced this issue fairly consistently (~1 out of 4) on both boot2docker and a ubuntu 14.04 host. Both run docker 1.4.1 and registry 0.9.1. The same problem happened in earlier versions of docker and registry as well.

I used the default configurations of both docker and registry.

The script that triggered this issue creates a registry container:

$ docker create -P --name registry-container registry

And then repeats these steps over and over again:

  1. docker start registry-container
  2. on localhost, push ~30 containers to registry
  3. on a remote host, pull all those containers from registry
  4. docker stop registry-container
  5. go to step 1

Let me know if there's more info I can provide.

@cinterloper

This comment has been minimized.

Show comment
Hide comment
@cinterloper

cinterloper Feb 3, 2015

I had this issue on Amazon Linux.
It started for me after i upgraded docker.

I resolved the issue by running 'docker pull registry' , which updated the copy of the registry image

cinterloper commented Feb 3, 2015

I had this issue on Amazon Linux.
It started for me after i upgraded docker.

I resolved the issue by running 'docker pull registry' , which updated the copy of the registry image

@jcockhren

This comment has been minimized.

Show comment
Hide comment
@jcockhren

jcockhren Feb 6, 2015

I can confirm @cinterloper findings for Ubuntu x64 14.04.1

jcockhren commented Feb 6, 2015

I can confirm @cinterloper findings for Ubuntu x64 14.04.1

@akaspin

This comment has been minimized.

Show comment
Hide comment
@akaspin

akaspin Feb 12, 2015

Confirm. EC2, CoreOS.

akaspin commented Feb 12, 2015

Confirm. EC2, CoreOS.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Feb 12, 2015

Member

People: if you have that issue, please don't add more comments unless the information does add value.

Adding -e GUNICORN_OPTS=["--preload"] to your docker run command will workaround the issue as a short-term remediation.

Thanks.

Member

dmp42 commented Feb 12, 2015

People: if you have that issue, please don't add more comments unless the information does add value.

Adding -e GUNICORN_OPTS=["--preload"] to your docker run command will workaround the issue as a short-term remediation.

Thanks.

@bunop

This comment has been minimized.

Show comment
Hide comment
@bunop

bunop Mar 9, 2015

Hi all, This bug is due to line 330 in /usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py, in which there is a "os.remove(lock_path)" and this lock file doesn't exists using the SETTINGS_FLAVOR=dev and a local storage path, as stated by user guide. I try to fix it by removing file after testing for its existance:

        finally:
            #check for file existance
            if os.path.exists(lock_path):
                #the original 330 line of /usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py
                os.remove(lock_path)
        return result

And then docker-registry will start without the GUNICORN_OPTS

bunop commented Mar 9, 2015

Hi all, This bug is due to line 330 in /usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py, in which there is a "os.remove(lock_path)" and this lock file doesn't exists using the SETTINGS_FLAVOR=dev and a local storage path, as stated by user guide. I try to fix it by removing file after testing for its existance:

        finally:
            #check for file existance
            if os.path.exists(lock_path):
                #the original 330 line of /usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py
                os.remove(lock_path)
        return result

And then docker-registry will start without the GUNICORN_OPTS

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Mar 9, 2015

Member

@bunop this has nothing to do with local storage.
The lock file always exist no matter the storage, as long as you enable the search index.

Member

dmp42 commented Mar 9, 2015

@bunop this has nothing to do with local storage.
The lock file always exist no matter the storage, as long as you enable the search index.

AntonioMeireles added a commit to AntonioMeireles/kubernetes-vagrant-coreos-cluster that referenced this issue Mar 17, 2015

adding workaround for docker/docker-registry#892
Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes-vagrant-coreos-cluster that referenced this issue Mar 17, 2015

adding workaround for docker/docker-registry#892
Signed-off-by: António Meireles <antonio.meireles@reformi.st>

pires added a commit to pires/kubernetes-vagrant-coreos-cluster that referenced this issue Mar 17, 2015

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 17, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 17, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 17, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 17, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 18, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 18, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 19, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

AntonioMeireles added a commit to AntonioMeireles/kubernetes that referenced this issue Mar 20, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>

brendandburns added a commit to kubernetes/kubernetes that referenced this issue Mar 23, 2015

zutherb pushed a commit to zutherb/kubernetes-vagrant-coreos-cluster that referenced this issue Mar 28, 2015

Bernd Zuther
Merge branch 'master' of git://github.com/pires/kubernetes-vagrant-co…
…reos-cluster into pires-master

* 'master' of git://github.com/pires/kubernetes-vagrant-coreos-cluster: (35 commits)
  Fixes #51
  Allow for running privileged containers.
  Warning, only Kubernetes 0.13.x or newer is currently supported (and working).
  Removing etcd debugging stuff in order to simplify things.
  Ignoring bad SSL certificates when downloading kubectl.
  Fixing typos.
  Kubernetes upgraded to 0.13.2.
  kube-proxy is deprecating flag --ectd_servers in favor of --master. Refs #50
  Reverted to Kubernetes 0.12.2 since 0.13.x has broken services iptables rules. See kubernetes/kubernetes#5647
  Added waiter for Kubernetes API server before starting kube-proxy and kubelet.
  Latest Kubernetes version is really the latest tested, so 0.13.1 now.
  Desperate attempt at fixing so many issues that were brought by what we thought were smart moves. Sorry for that! Removing dockerized etcd 2.x Removing dockerized flannel Master node doesn't need to rely on its own dockercache for pulling images, for now. Refs #46 #47 #32
  Added more hypervisor related documentation love, namely removing DHCP from Virtualbox. Fixes #45
  Removed incompatible Docker flags because of bad merge.
  drop unneeded nfs related cruft.
  docker-cache container now restarts on failure. Refs #40
  only start docker in the minion nodes after the master's docker-cache is up.
  adding workaround for docker/docker-registry#892
  rollback 182252c. just default kubernetes to latest stable release.
  fix #36 - docker could start before flanneld
  ...

zutherb pushed a commit to zutherb/kubernetes-vagrant-coreos-cluster that referenced this issue Mar 28, 2015

Bernd Zuther
Merge branch 'pires-master'
* pires-master: (35 commits)
  Fixes #51
  Allow for running privileged containers.
  Warning, only Kubernetes 0.13.x or newer is currently supported (and working).
  Removing etcd debugging stuff in order to simplify things.
  Ignoring bad SSL certificates when downloading kubectl.
  Fixing typos.
  Kubernetes upgraded to 0.13.2.
  kube-proxy is deprecating flag --ectd_servers in favor of --master. Refs #50
  Reverted to Kubernetes 0.12.2 since 0.13.x has broken services iptables rules. See kubernetes/kubernetes#5647
  Added waiter for Kubernetes API server before starting kube-proxy and kubelet.
  Latest Kubernetes version is really the latest tested, so 0.13.1 now.
  Desperate attempt at fixing so many issues that were brought by what we thought were smart moves. Sorry for that! Removing dockerized etcd 2.x Removing dockerized flannel Master node doesn't need to rely on its own dockercache for pulling images, for now. Refs #46 #47 #32
  Added more hypervisor related documentation love, namely removing DHCP from Virtualbox. Fixes #45
  Removed incompatible Docker flags because of bad merge.
  drop unneeded nfs related cruft.
  docker-cache container now restarts on failure. Refs #40
  only start docker in the minion nodes after the master's docker-cache is up.
  adding workaround for docker/docker-registry#892
  rollback 182252c. just default kubernetes to latest stable release.
  fix #36 - docker could start before flanneld
  ...
@ashish235

This comment has been minimized.

Show comment
Hide comment
@ashish235

ashish235 Mar 30, 2015

I 'm still getting this error and unable to start the registry container. Going through the comments in this thread I couldn't derive a conclusion.

[root@localhost files]# docker run -p 5000:5000 registry
[info] POST /v1.15/containers/create
[8f91c94d] +job create()
[8f91c94d] +job log(create, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(create, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] -job create() = OK (0)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/attach?stderr=1&stdout=1&stream=1
[8f91c94d] +job container_inspect(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job container_inspect(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job attach(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/start
[8f91c94d] +job start(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] +job allocate_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job allocate_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job allocate_port(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job allocate_port(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job log(start, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(start, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] -job start(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[2015-03-30 07:06:34 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-03-30 07:06:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-03-30 07:06:34 +0000] [1] [INFO] Using worker: gevent
[2015-03-30 07:06:34 +0000] [12] [INFO] Booting worker with pid: 12
[2015-03-30 07:06:34 +0000] [13] [INFO] Booting worker with pid: 13
[2015-03-30 07:06:34 +0000] [14] [INFO] Booting worker with pid: 14
[2015-03-30 07:06:34 +0000] [15] [INFO] Booting worker with pid: 15
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:36 +0000 WARNING: Another process is creating the search database
30/Mar/2015:07:06:36 +0000 WARNING: Another process is creating the search database
30/Mar/2015:07:06:36 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-03-30 07:06:36 +0000] [14] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-03-30 07:06:36 +0000] [14] [INFO] Worker exiting (pid: 14)
[2015-03-30 07:06:37 +0000] [15] [INFO] Worker exiting (pid: 15)
[2015-03-30 07:06:37 +0000] [13] [INFO] Worker exiting (pid: 13)
[2015-03-30 07:06:37 +0000] [12] [INFO] Worker exiting (pid: 12)
[2015-03-30 07:06:37 +0000] [1] [INFO] Shutting down: Master
[2015-03-30 07:06:37 +0000] [1] [INFO] Reason: Worker failed to boot.
[8f91c94d] +job log(die, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(die, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] +job release_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job attach(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/wait
[8f91c94d] +job wait(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job release_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] -job wait(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)

Please someone help.

ashish235 commented Mar 30, 2015

I 'm still getting this error and unable to start the registry container. Going through the comments in this thread I couldn't derive a conclusion.

[root@localhost files]# docker run -p 5000:5000 registry
[info] POST /v1.15/containers/create
[8f91c94d] +job create()
[8f91c94d] +job log(create, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(create, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] -job create() = OK (0)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/attach?stderr=1&stdout=1&stream=1
[8f91c94d] +job container_inspect(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job container_inspect(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job attach(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/start
[8f91c94d] +job start(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] +job allocate_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job allocate_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job allocate_port(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job allocate_port(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] +job log(start, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(start, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] -job start(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[2015-03-30 07:06:34 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-03-30 07:06:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-03-30 07:06:34 +0000] [1] [INFO] Using worker: gevent
[2015-03-30 07:06:34 +0000] [12] [INFO] Booting worker with pid: 12
[2015-03-30 07:06:34 +0000] [13] [INFO] Booting worker with pid: 13
[2015-03-30 07:06:34 +0000] [14] [INFO] Booting worker with pid: 14
[2015-03-30 07:06:34 +0000] [15] [INFO] Booting worker with pid: 15
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: Cache storage disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 WARNING: LRU cache disabled!
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:35 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
30/Mar/2015:07:06:36 +0000 WARNING: Another process is creating the search database
30/Mar/2015:07:06:36 +0000 WARNING: Another process is creating the search database
30/Mar/2015:07:06:36 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-03-30 07:06:36 +0000] [14] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-03-30 07:06:36 +0000] [14] [INFO] Worker exiting (pid: 14)
[2015-03-30 07:06:37 +0000] [15] [INFO] Worker exiting (pid: 15)
[2015-03-30 07:06:37 +0000] [13] [INFO] Worker exiting (pid: 13)
[2015-03-30 07:06:37 +0000] [12] [INFO] Worker exiting (pid: 12)
[2015-03-30 07:06:37 +0000] [1] [INFO] Shutting down: Master
[2015-03-30 07:06:37 +0000] [1] [INFO] Reason: Worker failed to boot.
[8f91c94d] +job log(die, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest)
[8f91c94d] -job log(die, 2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd, registry:latest) = OK (0)
[8f91c94d] +job release_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job attach(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[info] POST /v1.15/containers/2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd/wait
[8f91c94d] +job wait(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd)
[8f91c94d] -job release_interface(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)
[8f91c94d] -job wait(2f6d143842dd4a86c7f9bfdd3eb92b0754291abfde5d9ddc8ee0d31a1475b5bd) = OK (0)

Please someone help.

akram added a commit to akram/kubernetes that referenced this issue Apr 7, 2015

workaround for docker/docker-registry#892 /crasher on CoreOS' cloud-c…
…onfigs.

avoids race conditions (docker/docker-registry#892) when multiple workers are
attempting to run sql migrations simultaneously. dunno why this isn't the
registry's default since this isn't specially hard to trigger so we're just
playing safe here.

Signed-off-by: António Meireles <antonio.meireles@reformi.st>
@kulbida

This comment has been minimized.

Show comment
Hide comment
@kulbida

kulbida Apr 13, 2015

I'm having the same crash here...

kulbida commented Apr 13, 2015

I'm having the same crash here...

@kulbida

This comment has been minimized.

Show comment
Hide comment
@kulbida

kulbida Apr 13, 2015

2015-04-13 02:39:15 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-04-13 02:39:15 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-04-13 02:39:15 +0000] [1] [INFO] Using worker: gevent
[2015-04-13 02:39:15 +0000] [12] [INFO] Booting worker with pid: 12
[2015-04-13 02:39:15 +0000] [13] [INFO] Booting worker with pid: 13
[2015-04-13 02:39:15 +0000] [14] [INFO] Booting worker with pid: 14
[2015-04-13 02:39:15 +0000] [15] [INFO] Booting worker with pid: 15
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 WARNING: Another process is creating the search database
13/Apr/2015:02:39:16 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-04-13 02:39:16 +0000] [13] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-04-13 02:39:16 +0000] [13] [INFO] Worker exiting (pid: 13)
13/Apr/2015:02:39:16 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-04-13 02:39:16 +0000] [15] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-04-13 02:39:16 +0000] [15] [INFO] Worker exiting (pid: 15)
[2015-04-13 02:39:16 +0000] [12] [INFO] Worker exiting (pid: 12)
[2015-04-13 02:39:17 +0000] [14] [INFO] Worker exiting (pid: 14)
[2015-04-13 02:39:17 +0000] [1] [INFO] Shutting down: Master
[2015-04-13 02:39:17 +0000] [1] [INFO] Reason: Worker failed to boot.

kulbida commented Apr 13, 2015

2015-04-13 02:39:15 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-04-13 02:39:15 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-04-13 02:39:15 +0000] [1] [INFO] Using worker: gevent
[2015-04-13 02:39:15 +0000] [12] [INFO] Booting worker with pid: 12
[2015-04-13 02:39:15 +0000] [13] [INFO] Booting worker with pid: 13
[2015-04-13 02:39:15 +0000] [14] [INFO] Booting worker with pid: 14
[2015-04-13 02:39:15 +0000] [15] [INFO] Booting worker with pid: 15
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 WARNING: Cache storage disabled!
13/Apr/2015:02:39:15 +0000 WARNING: LRU cache disabled!
13/Apr/2015:02:39:15 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
13/Apr/2015:02:39:15 +0000 WARNING: Another process is creating the search database
13/Apr/2015:02:39:16 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-04-13 02:39:16 +0000] [13] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-04-13 02:39:16 +0000] [13] [INFO] Worker exiting (pid: 13)
13/Apr/2015:02:39:16 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-04-13 02:39:16 +0000] [15] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-04-13 02:39:16 +0000] [15] [INFO] Worker exiting (pid: 15)
[2015-04-13 02:39:16 +0000] [12] [INFO] Worker exiting (pid: 12)
[2015-04-13 02:39:17 +0000] [14] [INFO] Worker exiting (pid: 14)
[2015-04-13 02:39:17 +0000] [1] [INFO] Shutting down: Master
[2015-04-13 02:39:17 +0000] [1] [INFO] Reason: Worker failed to boot.

@kulbida

This comment has been minimized.

Show comment
Hide comment
@kulbida

kulbida Apr 13, 2015

Used -e GUNICORN_OPTS=["--preload"] as a temp workaround...

kulbida commented Apr 13, 2015

Used -e GUNICORN_OPTS=["--preload"] as a temp workaround...

Hikariii pushed a commit to Hikariii/kubernetes-vagrant-coreos-cluster that referenced this issue May 1, 2015

adding workaround for docker/docker-registry#892
Signed-off-by: António Meireles <antonio.meireles@reformi.st>

Hikariii pushed a commit to Hikariii/kubernetes-vagrant-coreos-cluster that referenced this issue May 1, 2015

@wptad

This comment has been minimized.

Show comment
Hide comment
@wptad

wptad May 6, 2015

-e GUNICORN_OPTS=["--preload"] works for me

wptad commented May 6, 2015

-e GUNICORN_OPTS=["--preload"] works for me

@charlielin

This comment has been minimized.

Show comment
Hide comment
@charlielin

charlielin May 7, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

charlielin commented May 7, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

@joda70

This comment has been minimized.

Show comment
Hide comment
@joda70

joda70 May 8, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

joda70 commented May 8, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

@ashish235

This comment has been minimized.

Show comment
Hide comment
@ashish235

ashish235 May 8, 2015

Works for me every time. I use -e GUNICORN_OPTS=["--preload"] by default now to start registry.

ashish235 commented May 8, 2015

Works for me every time. I use -e GUNICORN_OPTS=["--preload"] by default now to start registry.

@patjlm

This comment has been minimized.

Show comment
Hide comment
@patjlm

patjlm May 13, 2015

Same here. -e GUNICORN_OPTS=["--preload"] made the trick

patjlm commented May 13, 2015

Same here. -e GUNICORN_OPTS=["--preload"] made the trick

@mayaguang

This comment has been minimized.

Show comment
Hide comment
@mayaguang

mayaguang May 16, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

mayaguang commented May 16, 2015

-e GUNICORN_OPTS=["--preload"] works for me, too

craigfurman added a commit to cloudfoundry-attic/garden-linux that referenced this issue Jun 4, 2015

Fix flake in docker registry test
Docker registry sometimes fails to start, see docker/docker-registry#892

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>

craigfurman added a commit to cloudfoundry-attic/garden-linux that referenced this issue Jun 5, 2015

Fix flake in docker registry test
Docker registry sometimes fails to start, see docker/docker-registry#892

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>

craigfurman added a commit to cloudfoundry-attic/garden-linux that referenced this issue Jun 9, 2015

Fix flake in docker registry test
Docker registry sometimes fails to start, see docker/docker-registry#892

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>

craigfurman added a commit to cloudfoundry-attic/garden-linux that referenced this issue Jun 9, 2015

Fix flake in docker registry test
Docker registry sometimes fails to start, see docker/docker-registry#892

Signed-off-by: Julian Friedman <julz.friedman@uk.ibm.com>
@dalanlan

This comment has been minimized.

Show comment
Hide comment
@dalanlan

dalanlan Jul 16, 2015

Same pb still.

dalanlan commented Jul 16, 2015

Same pb still.

@felixgao

This comment has been minimized.

Show comment
Hide comment
@felixgao

felixgao Jul 23, 2015

Using the latest registry from docker and still having the same problem

docker logs private_registry
[2015-07-23 21:47:47 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-07-23 21:47:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-07-23 21:47:47 +0000] [1] [INFO] Using worker: gevent
[2015-07-23 21:47:47 +0000] [13] [INFO] Booting worker with pid: 13
[2015-07-23 21:47:48 +0000] [14] [INFO] Booting worker with pid: 14
[2015-07-23 21:47:48 +0000] [15] [INFO] Booting worker with pid: 15
[2015-07-23 21:47:48 +0000] [22] [INFO] Booting worker with pid: 22
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-07-23 21:47:48 +0000] [22] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-07-23 21:47:48 +0000] [22] [INFO] Worker exiting (pid: 22)
23/Jul/2015:21:47:48 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-07-23 21:47:48 +0000] [13] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-07-23 21:47:48 +0000] [13] [INFO] Worker exiting (pid: 13)
Traceback (most recent call last):
File "/usr/local/bin/gunicorn", line 11, in
sys.exit(run())
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 185, in run
super(Application, self).run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 71, in run
Arbiter(self).run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 196, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 292, in halt
self.stop()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 343, in stop
time.sleep(0.1)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 209, in handle_chld
self.reap_workers()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
[2015-07-23 21:47:50 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-07-23 21:47:50 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-07-23 21:47:50 +0000] [1] [INFO] Using worker: gevent
[2015-07-23 21:47:50 +0000] [14] [INFO] Booting worker with pid: 14
[2015-07-23 21:47:50 +0000] [15] [INFO] Booting worker with pid: 15
[2015-07-23 21:47:50 +0000] [18] [INFO] Booting worker with pid: 18
[2015-07-23 21:47:50 +0000] [23] [INFO] Booting worker with pid: 23
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage

felixgao commented Jul 23, 2015

Using the latest registry from docker and still having the same problem

docker logs private_registry
[2015-07-23 21:47:47 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-07-23 21:47:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-07-23 21:47:47 +0000] [1] [INFO] Using worker: gevent
[2015-07-23 21:47:47 +0000] [13] [INFO] Booting worker with pid: 13
[2015-07-23 21:47:48 +0000] [14] [INFO] Booting worker with pid: 14
[2015-07-23 21:47:48 +0000] [15] [INFO] Booting worker with pid: 15
[2015-07-23 21:47:48 +0000] [22] [INFO] Booting worker with pid: 22
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:48 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:48 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:48 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-07-23 21:47:48 +0000] [22] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-07-23 21:47:48 +0000] [22] [INFO] Worker exiting (pid: 22)
23/Jul/2015:21:47:48 +0000 WARNING: DB is disconnected. Reconnect to it.
[2015-07-23 21:47:48 +0000] [13] [ERROR] Exception in worker process:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 507, in spawn_worker
worker.init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/ggevent.py", line 193, in init_process
super(GeventWorker, self).init_process()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/workers/base.py", line 114, in init_process
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 66, in wsgi
self.callable = self.load()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 65, in load
return self.load_wsgiapp()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/util.py", line 356, in import_app
import(module)
File "/usr/local/lib/python2.7/dist-packages/docker_registry/wsgi.py", line 27, in
from .search import * # noqa
File "/usr/local/lib/python2.7/dist-packages/docker_registry/search.py", line 14, in
INDEX = index.load(cfg.search_backend.lower())
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/init.py", line 82, in load
return db.SQLAlchemyIndex()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/lib/index/db.py", line 86, in init
self._setup_database()
File "/usr/local/lib/python2.7/dist-packages/docker_registry/toolkit.py", line 330, in wrapper
os.remove(lock_path)
OSError: [Errno 2] No such file or directory: './registry._setup_database.lock'
[2015-07-23 21:47:48 +0000] [13] [INFO] Worker exiting (pid: 13)
Traceback (most recent call last):
File "/usr/local/bin/gunicorn", line 11, in
sys.exit(run())
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/wsgiapp.py", line 74, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 185, in run
super(Application, self).run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/app/base.py", line 71, in run
Arbiter(self).run()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 196, in run
self.halt(reason=inst.reason, exit_status=inst.exit_status)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 292, in halt
self.stop()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 343, in stop
time.sleep(0.1)
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 209, in handle_chld
self.reap_workers()
File "/usr/local/lib/python2.7/dist-packages/gunicorn/arbiter.py", line 459, in reap_workers
raise HaltServer(reason, self.WORKER_BOOT_ERROR)
gunicorn.errors.HaltServer: <HaltServer 'Worker failed to boot.' 3>
[2015-07-23 21:47:50 +0000] [1] [INFO] Starting gunicorn 19.1.1
[2015-07-23 21:47:50 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2015-07-23 21:47:50 +0000] [1] [INFO] Using worker: gevent
[2015-07-23 21:47:50 +0000] [14] [INFO] Booting worker with pid: 14
[2015-07-23 21:47:50 +0000] [15] [INFO] Booting worker with pid: 15
[2015-07-23 21:47:50 +0000] [18] [INFO] Booting worker with pid: 18
[2015-07-23 21:47:50 +0000] [23] [INFO] Booting worker with pid: 23
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage
23/Jul/2015:21:47:50 +0000 WARNING: Cache storage disabled!
23/Jul/2015:21:47:50 +0000 WARNING: LRU cache disabled!
23/Jul/2015:21:47:50 +0000 DEBUG: Will return docker-registry.drivers.file.Storage

@dalanlan

This comment has been minimized.

Show comment
Hide comment
@dalanlan

dalanlan Jul 24, 2015

Uhh..This prj has been deprecated. You should switch to v2 instead.

dalanlan commented Jul 24, 2015

Uhh..This prj has been deprecated. You should switch to v2 instead.

@ashish235

This comment has been minimized.

Show comment
Hide comment
@ashish235

ashish235 Jul 27, 2015

For some strange reason, I feel the the v2 is slower than v1. The same image used to take 45 secs on my v1 registry to push while on v2 it's almost 2 mins. I 'm not sure why this delay.

Anyone facing same issue?

ashish235 commented Jul 27, 2015

For some strange reason, I feel the the v2 is slower than v1. The same image used to take 45 secs on my v1 registry to push while on v2 it's almost 2 mins. I 'm not sure why this delay.

Anyone facing same issue?

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jul 27, 2015

Member

@ashish235 the security model and ACL model are different.
v2 is optimized for security and pull performance, so, push might be slower depending on the scenario (although there is work ongoing to optimize that).

Member

dmp42 commented Jul 27, 2015

@ashish235 the security model and ACL model are different.
v2 is optimized for security and pull performance, so, push might be slower depending on the scenario (although there is work ongoing to optimize that).

@andrask

This comment has been minimized.

Show comment
Hide comment
@andrask

andrask Aug 4, 2015

@dalanlan Unfortunately, v2 has several drawbacks when used in a CI pipeline (with plenty of network bandwidth and storage):

  1. push is slow
  2. layers are pushed repeatedly due to a bug
  3. Additionally, there is no way to browse a v2 registry, thus, UIs don't exist.
    These provide enough reason, for example, for us to stick with v1 until they get solved.

andrask commented Aug 4, 2015

@dalanlan Unfortunately, v2 has several drawbacks when used in a CI pipeline (with plenty of network bandwidth and storage):

  1. push is slow
  2. layers are pushed repeatedly due to a bug
  3. Additionally, there is no way to browse a v2 registry, thus, UIs don't exist.
    These provide enough reason, for example, for us to stick with v1 until they get solved.
@dalanlan

This comment has been minimized.

Show comment
Hide comment
@dalanlan

dalanlan Aug 4, 2015

Fair enough. The progress of v2 is getting a bit slow, i'll admit.

dalanlan commented Aug 4, 2015

Fair enough. The progress of v2 is getting a bit slow, i'll admit.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Aug 6, 2015

Member

@andrask

  1. there is work going on to enhance that, and also ways to optimize it depending on your use case - if you can elaborate a bit on your specific usage, there may be low hanging fruits for you
  2. do you have any specific issue number for this one? would like to look into it
  3. there are plenty of existing UI for v2, including search (built on top of the notification API). Portus comes to mind for example, or DTR. Furthermore, the catalog API has been merged, unlocking advanced tooling/search to be built.

@dalanlan
Registry 2.1 release candidate have just been cut.
You will find the release notes here:
https://github.com/docker/distribution/releases/tag/v2.1.0-rc.0

and the full list of closed issues and merged PRs (about 150 PRs merged for this release) is here: https://github.com/docker/distribution/issues?utf8=%E2%9C%93&q=milestone%3ARegistry%2F2.1+is%3Aclosed+

Since development for 2.2 will start soon, your feedback is welcome on what still needs to be addressed.

Member

dmp42 commented Aug 6, 2015

@andrask

  1. there is work going on to enhance that, and also ways to optimize it depending on your use case - if you can elaborate a bit on your specific usage, there may be low hanging fruits for you
  2. do you have any specific issue number for this one? would like to look into it
  3. there are plenty of existing UI for v2, including search (built on top of the notification API). Portus comes to mind for example, or DTR. Furthermore, the catalog API has been merged, unlocking advanced tooling/search to be built.

@dalanlan
Registry 2.1 release candidate have just been cut.
You will find the release notes here:
https://github.com/docker/distribution/releases/tag/v2.1.0-rc.0

and the full list of closed issues and merged PRs (about 150 PRs merged for this release) is here: https://github.com/docker/distribution/issues?utf8=%E2%9C%93&q=milestone%3ARegistry%2F2.1+is%3Aclosed+

Since development for 2.2 will start soon, your feedback is welcome on what still needs to be addressed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.