New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Dockerfile with recent Sqlite + Spatialite #280

Merged
merged 2 commits into from May 23, 2018

Conversation

Projects
None yet
3 participants
@r4vi
Contributor

r4vi commented May 22, 2018

This solves #278 without bloating the Dockerfile too much, the image size is now
495MB (original was ~240MB) but it could be reduced significantly if we only
copied the output of the compilation of spatialite and friends to
/usr/local/lib, instead of the entirety of it however that will take more time.

In the python code change references to import sqlite3 to import pysqlite3
and it should use the compiled version of sqlite3.23.1. You don't need to
try/except because pysqlite3 falls back to builtin sqlite3 if there is no
compiled version.

  $ docker run --rm -it datasette spatialite
  SpatiaLite version ..: 4.4.0-RC0	Supported Extensions:
    - 'VirtualShape'	[direct Shapefile access]
    - 'VirtualDbf'		[direct DBF access]
    - 'VirtualXL'		[direct XLS access]
    - 'VirtualText'		[direct CSV/TXT access]
    - 'VirtualNetwork'	[Dijkstra shortest path]
    - 'RTree'		[Spatial Index - R*Tree]
    - 'MbrCache'		[Spatial Index - MBR cache]
    - 'VirtualSpatialIndex'	[R*Tree metahandler]
    - 'VirtualElementary'	[ElemGeoms metahandler]
    - 'VirtualKNN'	[K-Nearest Neighbors metahandler]
    - 'VirtualXPath'	[XML Path Language - XPath]
    - 'VirtualFDO'		[FDO-OGR interoperability]
    - 'VirtualGPKG'	[OGC GeoPackage interoperability]
    - 'VirtualBBox'		[BoundingBox tables]
    - 'SpatiaLite'		[Spatial SQL - OGC]
  PROJ.4 version ......: Rel. 4.9.3, 15 August 2016
  GEOS version ........: 3.5.1-CAPI-1.9.1 r4246
  TARGET CPU ..........: x86_64-linux-gnu
  the SPATIAL_REF_SYS table already contains some row(s)
  SQLite version ......: 3.23.1
  Enter ".help" for instructions
  SQLite version 3.23.1 2018-04-10 17:39:29
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  spatialite>
$ docker run --rm -it datasette python -c "import pysqlite3; print(pysqlite3.sqlite_version)"
3.23.1
Build Dockerfile with recent Sqlite + Spatialite
This solves #278 without bloating the Dockerfile too much, the image size is now
495MB (original was ~300MB) but it could be reduced significantly if we only
copied the output of the compilation of spatialite and friends to
/usr/local/lib, instead of the entirety of it however that will take more time.

In the python code change references to `import sqlite3` to `import pysqlite3`
and it should use the compiled version of sqlite3.23.1. You don't need to
try/except because pysqlite3 falls back to builtin sqlite3 if there is no
compiled version.

```bash
  $ docker run --rm -it datasette spatialite
  SpatiaLite version ..: 4.4.0-RC0	Supported Extensions:
    - 'VirtualShape'	[direct Shapefile access]
    - 'VirtualDbf'		[direct DBF access]
    - 'VirtualXL'		[direct XLS access]
    - 'VirtualText'		[direct CSV/TXT access]
    - 'VirtualNetwork'	[Dijkstra shortest path]
    - 'RTree'		[Spatial Index - R*Tree]
    - 'MbrCache'		[Spatial Index - MBR cache]
    - 'VirtualSpatialIndex'	[R*Tree metahandler]
    - 'VirtualElementary'	[ElemGeoms metahandler]
    - 'VirtualKNN'	[K-Nearest Neighbors metahandler]
    - 'VirtualXPath'	[XML Path Language - XPath]
    - 'VirtualFDO'		[FDO-OGR interoperability]
    - 'VirtualGPKG'	[OGC GeoPackage interoperability]
    - 'VirtualBBox'		[BoundingBox tables]
    - 'SpatiaLite'		[Spatial SQL - OGC]
  PROJ.4 version ......: Rel. 4.9.3, 15 August 2016
  GEOS version ........: 3.5.1-CAPI-1.9.1 r4246
  TARGET CPU ..........: x86_64-linux-gnu
  the SPATIAL_REF_SYS table already contains some row(s)
  SQLite version ......: 3.23.1
  Enter ".help" for instructions
  SQLite version 3.23.1 2018-04-10 17:39:29
  Enter ".help" for instructions
  Enter SQL statements terminated with a ";"
  spatialite>
```

```bash
$ docker run --rm -it datasette python -c "import pysqlite3; print(pysqlite3.sqlite_version)"
3.23.1
```
@r4vi

This comment has been minimized.

Contributor

r4vi commented May 22, 2018

>>> import sqlite3
>>> sqlite3.sqlite_version
'3.23.1'
>>> 

running the above in the container seems to show 3.23.1 too so maybe we don't need pysqlite3 at all?

@simonw

This comment has been minimized.

Owner

simonw commented May 22, 2018

This looks amazing! Can't wait to try this out this evening.

@simonw

This comment has been minimized.

Owner

simonw commented May 22, 2018

Yeah let's try this without pysqlite3 and see if we still get the correct version.

@r4vi

This comment has been minimized.

Contributor

r4vi commented May 22, 2018

@simonw

This comment has been minimized.

Owner

simonw commented May 23, 2018

I grabbed just your Dockerfile and built it like this:

docker build . -t datasette

Once it had built, I ran it like this:

docker run -p 8001:8001 -v `pwd`:/mnt datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
        --load-extension=/usr/local/lib/mod_spatialite.so

(The fixtures.db file is created by running python tests/fixtures.py fixtures.db)

Then I visited http://localhost:8001/-/versions and I got this:

{
    "datasette": {
        "version": "0+unknown"
    },
    "python": {
        "full": "3.6.3 (default, Dec 12 2017, 06:37:05) \n[GCC 6.3.0 20170516]",
        "version": "3.6.3"
    },
    "sqlite": {
        "extensions": {
            "json1": null,
            "spatialite": "4.4.0-RC0"
        },
        "fts_versions": [
            "FTS4",
            "FTS3"
        ],
        "version": "3.23.1"
    }
}

Fantastic! I'm getting SQLite 3.23.1 and SpatiaLite 4.4.0-RC0

setup.py Outdated
@@ -38,6 +38,7 @@ def get_version():
'hupper==1.0',
'pint==0.8.1',
'pluggy>=0.1.0,<1.0',
'pysqlite3==0.1.4',

This comment has been minimized.

@simonw

simonw May 23, 2018

Owner

This is definitely unnecessary, it looks like your Dockerfile is already building the most recent version of SQLite.

Dockerfile Outdated
RUN wget "https://www.gaia-gis.it/gaia-sins/spatialite-tools-4.4.0-RC0.tar.gz" && tar zxvf spatialite-tools-4.4.0-RC0.tar.gz && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install
# Add local code to the image instead of fetching from pypi.
ADD . /datasette

This comment has been minimized.

@larsclaussen

larsclaussen May 23, 2018

Docker best practices:

Although ADD and COPY are functionally similar, generally speaking, COPY is preferred. That’s because it’s more transparent than ADD

Dockerfile Outdated
RUN apt install -y python3-dev gcc libsqlite3-mod-spatialite
RUN apt update \
&& apt install -y python3-dev build-essential wget libxml2 libxml2-dev libproj-dev libgeos-dev libsqlite3-dev zlib1g-dev pkg-config unzip \
&& apt clean

This comment has been minimized.

@larsclaussen

larsclaussen May 23, 2018

maybe && rm -rf /var/lib/apt/lists/* /tmp/* too?

Dockerfile Outdated
RUN wget "https://www.gaia-gis.it/gaia-sins/readosm-1.1.0.tar.gz" && tar zxvf readosm-1.1.0.tar.gz && cd readosm-1.1.0 && ./configure && make && make install
RUN wget "https://www.gaia-gis.it/gaia-sins/spatialite-tools-4.4.0-RC0.tar.gz" && tar zxvf spatialite-tools-4.4.0-RC0.tar.gz && cd spatialite-tools-4.4.0-RC0 && ./configure && make && make install

This comment has been minimized.

@larsclaussen

larsclaussen May 23, 2018

are the tar files removed after the installation? That will make the image smaller too

This comment has been minimized.

@r4vi

r4vi May 23, 2018

Contributor

@larsclaussen they aren't but they're part of the builder image not the app image so it shouldn't make a difference (same for && rm -rf /var/lib/apt/lists/* /tmp/*) - unless I'm not understanding the builder image mechanism correctly.

This comment has been minimized.

@larsclaussen

larsclaussen May 23, 2018

Ah..didn't catch the multi stage build. You're right, shouldn't make a difference then!

Dockerfile consistency and FTS5 detection
This makes sure the compile steps for the various libraries are done in only one
layer.

Also updates the query used to check for FTS5 as the old version wasn't
detecting FTS5 for some reason.
@r4vi

This comment has been minimized.

Contributor

r4vi commented May 23, 2018

Running:

docker run -p 8001:8001 -v `pwd`:/mnt datasette \
    datasette -p 8001 -h 0.0.0.0 /mnt/fixtures.db \
        --load-extension=/usr/local/lib/mod_spatialite.so

is now returning FTS5 enabled in the versions output:

{
    "datasette": {
        "version": "0.22"
    },
    "python": {
        "full": "3.6.5 (default, May  5 2018, 03:07:21) \n[GCC 6.3.0 20170516]",
        "version": "3.6.5"
    },
    "sqlite": {
        "extensions": {
            "json1": null,
            "spatialite": "4.4.0-RC0"
        },
        "fts_versions": [
            "FTS5",
            "FTS4",
            "FTS3"
        ],
        "version": "3.23.1"
    }
}

The old query didn't work because specifying (t TEXT) caused an error

@simonw

This comment has been minimized.

Owner

simonw commented May 23, 2018

@r4vi any objections to me merging this?

@r4vi

This comment has been minimized.

Contributor

r4vi commented May 23, 2018

@simonw simonw merged commit bd30c69 into simonw:master May 23, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@simonw

This comment has been minimized.

Owner

simonw commented May 23, 2018

Thank you very much! This is most excellent.

@simonw

This comment has been minimized.

Owner

simonw commented Jun 28, 2018

I pushed this to Docker Hub https://hub.docker.com/r/datasetteproject/datasette/ and added notes on how to use it to the documentation: http://datasette.readthedocs.io/en/latest/installation.html#using-docker

@psychemedia psychemedia referenced this pull request Nov 4, 2018

Open

Docker build tools #372

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment