Docker image: sqlite3.OperationalError: unable to open database file

[ghost]

Hi, I have followed the Docker instructions here:
https://github.com/commonsense/conceptnet5/wiki/Docker#building-conceptnet-with-docker
and this proceeded error-free.

I can run the container and see the following log:

$ docker run -it -p 10054 rspeer/conceptnet-web:5.4 -v /my-big-drive/data:/conceptnet_data
WARNING: Your kernel does not support memory swappiness capabilities, memory swappiness discarded.
[2015-10-13 21:13:57 +0000] [6] [INFO] Starting gunicorn 19.3.0
[2015-10-13 21:13:57 +0000] [6] [INFO] Listening at: http://0.0.0.0:10054 (6)
[2015-10-13 21:13:57 +0000] [6] [INFO] Using worker: sync
[2015-10-13 21:13:57 +0000] [9] [INFO] Booting worker with pid: 9
[2015-10-13 21:13:57 +0000] [11] [INFO] Booting worker with pid: 11
[2015-10-13 21:13:57 +0000] [13] [INFO] Booting worker with pid: 13
[2015-10-13 21:13:57 +0000] [15] [INFO] Booting worker with pid: 15

However if I then try an example query:
http://127.0.0.1:32768/data/5.4/c/en/toast

I get a 500 internal server error:

ERROR:conceptnet5:Exception on /data/5.4/c/en/toast [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.4/site-packages/Flask_Cors-2.1.0-py3.4.egg/flask_cors/extension.py", line 110, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.4/site-packages/Flask-0.10.1-py3.4.egg/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/src/conceptnet/conceptnet5/api.py", line 99, in query_node
    results = list(FINDER.lookup(path, offset=offset, limit=limit))
  File "/src/conceptnet/conceptnet5/query.py", line 75, in lookup
    self.load_index()
  File "/src/conceptnet/conceptnet5/query.py", line 58, in load_index
    self._db_filename, self._edge_dir, self.nshards
  File "/src/conceptnet/conceptnet5/formats/sql.py", line 211, in __init__
    self._connect()
  File "/src/conceptnet/conceptnet5/formats/sql.py", line 216, in _connect
    self.dbs[i] = sqlite3.connect(filename)
sqlite3.OperationalError: unable to open database file
xx.xx.xx.xx - - [13/Oct/2015:21:40:20 +0000] "GET /data/5.4/c/en/toast HTTP/1.1" 500 291 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.132 Safari/537.36"

-which looks like it may be the same underlying error as this:
#33

I have tried to figure out how to get a shell on the container so I can try the 'ln -s' fix, but have had no success.

Would appreciate any help or advice on
-how to access the container and try the above fix myself, or
-otherwise how to resolve the issue.

Thanks in advance for any advice!

[ghost]

Update (still not solved):

I have confirmed path and permissions for the /my-big-drive/data folder. I set ownership to $USER:docker and ensured rw access for user and group. I also tried variations of the -v /path/to/my-big-drive/data/ with no observable change in error message.

[rspeer]

Hm. I'm not an expert in Docker myself, but I think you might be able to find out what's going wrong if you could get a shell inside your container.

I have a script called 'dock' whose contents are:

#!/bin/bash
docker exec -it $@ /bin/bash

This lets me get a shell inside a container by typing 'sudo dock' and its container ID. You could also just substitute the container ID into that command. To see the IDs of running containers, run sudo docker ps.

[ghost]

Thanks for the reply. Nor me an expert in docker - I tend to use "real" vm's for manageability. I have tried:

$ docker exec -it <container-id> <command>

-where <command> is any cli command that might or might not exist, such as /bin/bash or "/bin/bash" or /bin/sh or busybox sh etc, etc. Obviously once /bin/bash fails, all of these are guesses, and all of these return exactly this (and nothing else):

no such file or directory

By issuing docker inspect <container-id> I can retrieve info about the container, and see for example that it provides a gateway on a private IP address, and by nmap-ing that I can discover that it is exposing various ports:

$ nmap xx.xx.xx.xx

Starting Nmap 6.00 ( http://nmap.org ) at 2015-10-16 13:31 EDT
Nmap scan report for xx.xx.xx.xx
Host is up (0.00040s latency).
Not shown: 996 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
111/tcp   open  rpcbind
32768/tcp open  filenet-tms

I can attempt to ssh into port 22 and receive a userid/password prompt, but of course I have no info on appropriate credentials.

Presumably someone must have built this image, and therefore have info on how to access the image? It is not sensible for me to keep guessing at how to access the image and using what credentials, that is not productive (to put it politely).

[rspeer]

I built the image. I can promise you that it has /bin/bash in it. I don't even know what's telling you "no such file or directory".

If I run this command:

docker exec -it web_conceptnet54_1 /bin/bash

I get a shell. Running export TERM=xterm in this shell makes it more functional, btw.

If I ask for a file that doesn't exist, such as /fake/bash, I get a more detailed line than the one you report:

2015/10/16 18:35:39 docker-exec: failed to exec: exec: "/fake/bash": stat /fake/bash: no such file or directory

The images were built with Docker 1.3 but I've been able to run them on 1.7.

I don't think trying to hack into Docker through open ports is going to be productive at all. I have my doubts that what you're reaching is even the container. It might be something else internal to Docker. The container does not open any ports except for 10054. It's not running an SSH server. It's not running any processes except for gunicorn and the Python processes it spawns.

The symlink that was necessary to patch a particular non-Docker version of 5.3 is not necessary.

Here's how I can confirm that the DB exists where it's supposed to inside the container:

$ python
>>> from conceptnet5.query import *
>>> get_data_filename('db/assertions.db.0')
'/conceptnet_data/db/assertions.db.0'
>>> import os
>>> os.path.exists('/conceptnet_data/db/assertions.db.0')
True
>>> query({'start': '/c/en/example'})[0]
{'sources': ['/s/rule/definition_section', '/s/rule/link_section', '/s/web/en.wiktionary.org/wiki/example'], 'rel': '/r/RelatedTo', 'dataset': '/d/wiktionary/en/en', 'context': '/ctx/all', 'end': '/c/en/pattern', 'surfaceStart': 'example', 'uri': '/a/[/r/RelatedTo/,/c/en/example/n/,/c/en/pattern/]', 'source_uri': '/or/[/and/[/s/rule/definition_section/,/s/web/en.wiktionary.org/wiki/example/]/,/and/[/s/rule/link_section/,/s/web/en.wiktionary.org/wiki/example/]/]', 'surfaceEnd': 'pattern', 'surfaceText': '[[example]] is related to [[pattern]]', 'start': '/c/en/example/n', 'license': '/l/CC/By-SA', 'features': ['/c/en/example/n /r/RelatedTo -', '/c/en/example/n - /c/en/pattern', '- /r/RelatedTo /c/en/pattern'], 'id': '/e/75cd7a28c6f260b57b2331a311020a6e013b42c0', 'weight': 1.5849625007211563}

So, all I can say is that something is extremely messed up about your filesystem in Docker. Not only is it not finding /bin/bash, it's not finding your /conceptnet_data volume either. If you're on a reasonably up-to-date version, you will probably need to get help from the Docker developers, not from me.

[ghost]

Thanks for the python snippet, that may be helpful in the future if I can ever get a cli. Here's versioning for the record:

Linux .. 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u1 x86_64 GNU/Linux
Docker version 1.8.2, build 0a8c2e3

What is confusing me is that this set up is all entirely default; the only exception being that I have placed the docker image in a non-standard location as suggested here, following the '-g option' instructions here. However, I am beginning to suspect that changing the docker image installation directory like that may not compatible with this image, meaning that it would not be possible to locate the docker image also on a /mnt/big-data/ drive. The only thing that is at all notable is that the message is exactly 'no such file or directory' with no capitalization, whereas the only OS message I can remember is 'No such file or directory' (capital N). Of course looking for 'no such file or directory' is worse than a needle in a haystack.

In case anyone else ends up in this dark place, here is a link to the docker forums post.

[ghost]

Update (not solved).

In case anyone is following this thread, I have posted docker daemon debug messages here

[0t2]

I have built docker image with Ubuntu 14.04, Docker version 1.9.0, build 76d6bc9

1. git clone https://github.com/commonsense/conceptnet5
2. • cd conceptnet5/lib/ && vi Dockerfile
   • change FROM python:3.4 to FROM python:3.5
   • docker build -t="conceptnet-base:5.4" .
3. • cd .. && vi Dockerfile
   • change FROM rspeer/conceptnet-base:5.4 to FROM conceptnet-base:5.4
   • vi setup.py and change 'flask' to'flask >= 0.10.1' on install_requires block. If you don't change it and build, you will get error: The 'flask' distribution was not found and is required by ConceptNet
   • docker build -t="conceptnet:5.4" .
4. • cd web && vi Dockerfile
   • change from FROM rspeer/conceptnet:5.4 to FROM conceptnet:5.4
   • docker build -t="conceptnet-web:5.4" .
5. docker run -p 10054:10054 --sig-proxy=false conceptnet-web:5.4 -v /root/data:/conceptnet_data

I hope this can help you.

[ghost]

Thank you for the update & instructions. I may try this at some point; though right now my plan when I can get back to this project is to build from scratch. For the record I have received exactly zero responses to my request for help on the docker forums; it seems their focus is solely on for-profit support contracts.

[EliasGitHub]

Dear @cwynd
I cannot understand what kind of query you are trying. I mean why are you using 32768 port???
It is supposed that the data can be accessed through 10054 port, isn't it?

However, I have the same problem I think. When I want to make a query, for example [http://localhost:10054/data/5.4/c/en/toast] it gives me this error
"Failed to connect to localhost port 10054: Connection refused"
And I am much tired of it..!!!

[ashutshar]

Facing same issue. Any solution found?