Docker installation fails to build due to apparent Solr schema version conflict #2255

beechnut · 2015-01-29T17:42:17Z

I followed the Docker installation instructions, and while the Solr and Postgresql containers are running, CKAN fails to build.

$ docker ps -a
CONTAINER ID        IMAGE                    COMMAND                CREATED             STATUS                              PORTS               NAMES
596bbfb621f2        ckan/solr:latest         "java -jar start.jar   5 days ago          Up 38 minutes                       8983/tcp            solr
b3af69f91938        ckan/postgresql:latest   "/usr/local/bin/run"   5 days ago          Up 42 minutes                       5432/tcp            db

Supporting services are running, but then (running it interactively to see logs):

$ docker run -it --rm -p 8000:8000 --link db:db --link solr:solr ckan/ckan
*** Running /etc/my_init.d/50_configure...
Distribution already installed:
  ckan 2.4a from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
  /etc/ckan/default/ckan.ini
Edited option sqlalchemy.url = "postgresql://ckan_default:pass@localhost/ckan_default"->"postgresql://ckan:ckan@172.17.0.40:5432/ckan" (section "app:main")
Option uncommented and set solr_url = "http://172.17.0.41:8983/solr/ckan" (section "app:main")
Option uncommented and set ckan.storage_path = "/var/lib/ckan" (section "app:main")
Edited option email_to = "you@yourdomain.com"->"disabled@example.com" (section "app:main")
Edited option error_email_from = "paste@localhost"->"ckan@5c6d3e2d141c" (section "app:main")
*** Running /etc/my_init.d/70_initdb...
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 204, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 144, in _load_config
    load_environment(conf.global_conf, conf.local_conf)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 232, in load_environment
    p.load_all(config)
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 124, in load_all
    unload_all()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 181, in unload_all
    unload(*reversed(_PLUGINS))
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 209, in unload
    plugins_update()
  File "/usr/lib/ckan/default/src/ckan/ckan/plugins/core.py", line 116, in plugins_update
    environment.update_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 270, in update_config
    search.check_solr_schema_version()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/search/__init__.py", line 306, in check_solr_schema_version
    % (version, ', '.join(SUPPORTED_SCHEMA_VERSIONS)))
ckan.lib.search.common.SearchError: SOLR schema version not supported: 2.0. Supported versions are [2.3]
*** /etc/my_init.d/70_initdb failed with status 1

*** Killing all processes...

Clearly there's a version conflict or a version string set incorrectly, but I'm not familiar enough with CKAN to understand how I might go about resolving this.

The text was updated successfully, but these errors were encountered:

mattva01 · 2015-01-30T06:52:21Z

It's because the image on docker hub is out of date. If you rebuild the solr docker image, it will work correctly (you may have to do some finagling as symlinks don't work with docker build)

milafrerichs · 2015-03-05T15:40:32Z

yes the ckan/solr image (https://registry.hub.docker.com/u/ckan/solr/) needs to be rebuild to use the new schema.xml updated here: #1883

I created a new ckan/solr image which you can use until the official image is updated:
https://registry.hub.docker.com/u/milafrerichs/ckan_solr/

gambl · 2015-04-02T16:28:25Z

Another approach using the current ckan/solr image is to run it and mount the new schema.xml into it:

The latest schema is at https://github.com/ckan/ckan/blob/master/ckan/config/solr/schema.xml

With the latest schema.xml in you current directory run ckan/solr as:

 docker run -d --name solr -v `pwd`/schema.xml:/opt/solr/example/solr/ckan/conf/schema.xml ckan/solr

filipefigcorreia · 2015-04-02T16:36:47Z

Another alternative is to create your own dockerfile that overrides schema.xml (check my comment at docker hub for an example).

SorraTheOrc · 2015-07-02T07:57:34Z

Thank you @milafrerichs for publishing a working container (and others for alternative workarounds). I'm looking forward to the official CKAN container being updated.

milafrerichs · 2015-07-02T08:29:01Z

You're welcome.
But maybe look at the official ckan docker repo:
https://github.com/ckan/ckan-docker

melantronic · 2015-10-01T14:58:41Z

Think Solr is still broken on the official one as just tried it. The schema fix posted on April 2 seemed to get further (i.e it managed to start up ckan) but then there are a lot of issues trying to use datasets. There are alot of internal servers errors pointing back to the SOLR issues.

beechnut · 2015-10-01T16:32:36Z

This project is so hard to use and even harder to maintain I've given up and am searching for other solutions. I recommend you do the same.

On Oct 1, 2015, at 7:58 AM, zemertz notifications@github.com wrote:

Think Solr is still broken on the official one as just tried it. The schema fix posted on April 2 seemed to get further (i.e it managed to start up ckan) but then there are a lot of issues trying to use datasets. There are alot of internal servers errors pointing back to the SOLR issues.

―
Reply to this email directly or view it on GitHub.

melantronic · 2015-10-02T08:10:26Z

Thanks for the heads up. Did you mean the docker/ckan or ckan itsself

Managed to get it working in the end (but needed to use milafrerichs image https://registry.hub.docker.com/u/milafrerichs/ckan_solr/ ).

Started up no errors, and then I needed to add an ADMIN like below with docker (NB:It will prompt to confirm user/password) - This is how you can then log in and add data sets to set things up.

docker run -i -t --link db:db --link solr:solr \ ckan/ckan \ /sbin/my_init -- \ /bin/bash -c \ '$CKAN_HOME/bin/paster --plugin=ckan sysadmin -c $CKAN_CONFIG/ckan.ini add administrator'

florianm · 2015-10-02T08:24:14Z

If you just need a working CKAN in docker containers, have you tried www.datacats.com? Did the trick for me and took away the solr headaches.

wardi · 2015-10-02T13:45:07Z

The open source ckan docker tools from datacats are now at https://github.com/datacats/datacats

davidread · 2015-10-07T11:49:24Z

There are several people complaining publicly elsewhere too:

https://hub.docker.com/r/ckan/ckan/
http://stackoverflow.com/questions/28796257/ckan-docker-container-not-running/32916857#32916857

I suggest we change our instructions to suggest people use Datacats: https://github.com/datacats/datacats because it's got some use from several people in the community and when issues arise they are looked at and solved quickly e.g. datacats/datacats#334

There was also the effort from @clementmouchet https://github.com/ckan/ckan-docker that looked good when it appeared last year, but that's had not any input for 6 months now.

florianm · 2015-10-07T12:55:12Z

+1 for datacats - always helpful and quick to fix.

clementmouchet · 2015-10-07T17:08:12Z

Hi there,
I don't have any opportunity to work on CKAN at work and I have indeed not been able to maintain the project over the past few months. Might be the right time to talk about either bringing it inline with datacats or archive it to avoid confusion.

Happy to discuss options with you.

vitorbaptista · 2015-12-02T17:17:55Z

I was bitten by this issue as well. Solved by using datacats instead 👍

gamesbook · 2016-01-05T09:23:29Z

@clementmouchet When you say "archive it", do you mean the whole CKAN project or just this Docker install? If the latter, will the datacats one then become the "de facto" install? In that case, can you please update the CKAN install page (http://docs.ckan.org/en/latest/maintaining/installing/install-using-docker.html) to redirect folk there instead of here -- right now we are all encountering the same issue (which gives a poor first impression of a major project).

davidread · 2016-01-05T11:09:45Z

I've now done a PR to stop people going down this bad route: #2821

gamesbook · 2016-01-05T12:56:04Z

@davidread Thanks! Unfortunately, the datacats approach does not work either.

Vanuan · 2016-01-27T23:03:51Z

So, does anybody has credentials to update (or remove) solr on Docker Hub?
https://hub.docker.com/r/ckan/solr/

davidread · 2016-01-28T11:34:09Z

@nickstenning has the credentials for https://hub.docker.com/r/ckan/ I believe

miguelbgouveia · 2016-04-22T08:08:36Z

For those how want use the docker image for solr that works with the ckan/ckan docker image:

docker run -d --name solr miguelbgouveia/solr-docker

And for any contribution:
https://github.com/miguelbgouveia/solr-docker

shabeermothi · 2016-11-21T19:36:56Z

@miguelbgouveia Isn't it
docker run -d --name solr miguelbgouveia/solr-docker ?

miguelbgouveia · 2016-11-22T09:20:45Z

You right @shabeermothi the command is:
docker run -d --name solr miguelbgouveia/solr-docker

Tks.

shabeermothi · 2016-11-22T16:29:54Z

Even after using solr image from miguelbgouveia/solr-docker or milafrerichs/ckan_solr i still get the same error.

Am following the below series of steps :
$ docker run -d --name db ckan/postgresql
$ docker run -d --name solr miguelbgouveia/solr-docker or docker run -d --name solr milafrerichs/ckan_solr
$ docker run -d -p 80:80 --link db:db --link solr:solr ckan/ckan

This is the error i get every time :

*** Running /etc/my_init.d/50_configure...
Distribution already installed:
  ckan 2.7.0a from /usr/lib/ckan/default/src/ckan
Creating /etc/ckan/default/ckan.ini
Now you should edit the config files
  /etc/ckan/default/ckan.ini
Edited option sqlalchemy.url = "postgresql://ckan_default:pass@localhost/ckan_default"->"postgresql://ckan:ckan@172.17.0.2:5432/ckan" (section "app:main")
Edited option ckan.site_url = ""->"http://192.168.0.6" (section "app:main")
Option uncommented and set solr_url = "http://172.17.0.3:8983/solr/ckan" (section "app:main")
Option uncommented and set ckan.storage_path = "/var/lib/ckan" (section "app:main")
Option uncommented and set email_to = "disabled@example.com" (section "app:main")
Option uncommented and set error_email_from = "ckan@f3e156017a3b" (section "app:main")
*** Running /etc/my_init.d/70_initdb...
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 230, in command
    self._load_config(cmd!='upgrade')
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 169, in _load_config
    conf = self._get_config()
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 166, in _get_config
    return appconfig('config:' + self.filename)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 261, in appconfig
    global_conf=global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 320, in _loadconfig
    return loader.get_context(object_type, name, global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 454, in get_context
    section)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 476, in _context_from_use
    object_type, name=use, global_conf=global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 406, in get_context
    global_conf=global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 296, in loadcontext
    global_conf=global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 328, in _loadegg
    return loader.get_context(object_type, name, global_conf)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 620, in get_context
    object_type, name=name)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 646, in find_egg_entry_point
    possible.append((entry.load(), protocol, entry.name))
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/pkg_resources.py", line 2048, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/__init__.py", line 9, in <module>
    from ckan.config.environment import load_environment
  File "/usr/lib/ckan/default/src/ckan/ckan/config/environment.py", line 17, in <module>
    import ckan.lib.helpers as helpers
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/helpers.py", line 33, in <module>
    import i18n
  File "/usr/lib/ckan/default/src/ckan/ckan/lib/i18n.py", line 52, in <module>
    import polib
ImportError: No module named polib
*** /etc/my_init.d/70_initdb failed with status 1

*** Killing all processes...

miguelbgouveia · 2016-11-22T16:59:24Z

I seams that is a problem only with ckan\ckan container and not with miguelbgouveia/solr-docker or milafrerichs/ckan_solr.

shabeermothi · 2016-11-22T18:08:29Z

What's the best way to run CKAN in docker ?

florianm · 2016-11-24T23:56:29Z

@shabeermothi since active development on datacats has ceased, Link Digital's AWS-based install looks tasty.
It is used in production on a larger Western Australian instance for which I had the honour of building the first CKAN deployment (based on datacats). I'd expect LD's setup to be more resilient against failure and scaling up better, as well as being more active and supported than datacats.

weschdc · 2016-12-01T16:51:00Z

I just went thru the docker build process and I am getting the same error message as others before: the last line complains that solr needs schema from [2.7].

what I don't understand is that the docker hub says all three repos (ckan/ckan, ckan/postgresql, & ckan/solr) were updated today. Is anybody (the contribitor) looking at the repos and making sure they are functioning repos ?

wardi · 2016-12-01T17:01:38Z

This should be updated to dev-2.7 https://github.com/ckan/ckan/blob/master/contrib/docker/solr/Dockerfile#L16

amercader · 2016-12-06T10:56:07Z

@wardi There is no dev-v2.7 yet. Do you mean master?

amercader · 2016-12-06T10:56:24Z

Looks like #3346 should address this

Conzar · 2017-04-04T02:13:05Z

This bug is also present in the packaged version of ckan and starting in ckan 2.5 and also in 2.6. The work-a-round that I found is to install ckan version 2.4 (packaged) and than install 2.5 or 2.6.

newkdukem · 2017-10-12T13:59:51Z

#3867 is causing Docker build to fail.
This Dockerfile takes the schame.xml from the master branch and with the referenced PR to master, is causing the build to fail.

rossjones · 2018-06-06T15:48:10Z

Is this all resolved by https://github.com/ckan/ckan/tree/master/contrib/docker now ?

davidread · 2018-06-06T16:14:38Z

Good spot. The 'install-from-docker' docs in the title has since been removed in favour of: http://docs.ckan.org/en/2.8/maintaining/installing/install-from-docker-compose.html so I'm closing.

Summary:

This method is dead: http://docs.ckan.org/en/latest/maintaining/installing/install-using-docker.html#installing-ckan
Datacats install is dead
This method works (at the time of writing): http://docs.ckan.org/en/2.8/maintaining/installing/install-from-docker-compose.html
https://hub.docker.com/r/ckan/ckan/ is now being updated again, but I don't have the details.

amercader self-assigned this Feb 3, 2015

milafrerichs mentioned this issue Mar 6, 2015

Docker compose #2338

Closed

davidread mentioned this issue Oct 7, 2015

Docker install broken #2677

Closed

davidread mentioned this issue Jan 5, 2016

Docker install just links to datacats #2820

Closed

Vanuan mentioned this issue Jan 27, 2016

Why there are two repos for ckan docker? ckan/ckan-docker#24

Closed

gerbyzation mentioned this issue Oct 31, 2017

Move docker-compose.yml to project root #3894

Closed

davidread closed this as completed Jun 6, 2018

Docker installation fails to build due to apparent Solr schema version conflict #2255

Docker installation fails to build due to apparent Solr schema version conflict #2255

Comments

beechnut commented Jan 29, 2015

mattva01 commented Jan 30, 2015

milafrerichs commented Mar 5, 2015

gambl commented Apr 2, 2015

filipefigcorreia commented Apr 2, 2015

SorraTheOrc commented Jul 2, 2015

milafrerichs commented Jul 2, 2015

melantronic commented Oct 1, 2015

beechnut commented Oct 1, 2015

melantronic commented Oct 2, 2015

florianm commented Oct 2, 2015

wardi commented Oct 2, 2015

davidread commented Oct 7, 2015

florianm commented Oct 7, 2015

clementmouchet commented Oct 7, 2015

vitorbaptista commented Dec 2, 2015

gamesbook commented Jan 5, 2016

davidread commented Jan 5, 2016

gamesbook commented Jan 5, 2016

Vanuan commented Jan 27, 2016

davidread commented Jan 28, 2016

miguelbgouveia commented Apr 22, 2016 • edited

shabeermothi commented Nov 21, 2016

miguelbgouveia commented Nov 22, 2016

shabeermothi commented Nov 22, 2016 • edited

miguelbgouveia commented Nov 22, 2016

shabeermothi commented Nov 22, 2016

florianm commented Nov 24, 2016

weschdc commented Dec 1, 2016

wardi commented Dec 1, 2016

amercader commented Dec 6, 2016

amercader commented Dec 6, 2016

Conzar commented Apr 4, 2017

newkdukem commented Oct 12, 2017

rossjones commented Jun 6, 2018

davidread commented Jun 6, 2018

miguelbgouveia commented Apr 22, 2016 •

edited

shabeermothi commented Nov 22, 2016 •

edited