519 update deployment docs second try #874

seanh · 2013-05-05T16:06:28Z

@nigelbabu I think you're probably the best person to review this, because you're doing the package install, and everything needs to be consistent between the package and source installs.

The new docs are built here:

I've tested these on my dev machine (using the symlinks into homedir tip) and on a fresh Ubuntu 12.04 VM (including deployment).

These files also changed:

After this is merged into master, I need to merge it into release-v2.0 and re-test. Some things are different between master and 2.0, e.g. test databases and user names. See #816

Todo:

Integrate Minor tweaks in Solr docs #814
Add postfix install instructions to deployment.rst, see [wip] [#519] Update deployment docs for CKAN 2.0 #821 (comment)
Integrate error using curl test after following datastore setup instructions #807
Fix Travis

Questions:

Is there a better user name than seanh?
I changed <VirtualHost 0.0.0.0:8080> to <VirtualHost 0.0.0.0:80> in the
apache sites-available file, because nginx isn't included in the from-source
deployment instructions. Does the package install include nginx? Can nginx
setup instructions be added to deployment.rst?
Can files like apache.wsgi and the apache sites-available file be shared
between source and package installs? Maybe add the file in git somewhere,
and make them both import it

Changes:

Changed default virtualenv location from ~/pyenv to /usr/lib/ckan/default
Changed default config files dir to /etc/ckan/default
Added tip for developers in source install docsS symlink /usr/lib/ckan and
/etc/ckan to directories in your homedir to install ckan in your homedir
Changed default database user from ckanuser to ckan_default
Changed default database from ckan_dev to ckan_default
Changed default datastore database from datastore to datastore_default
Changed default datastore database user from readonlyuser to datastore_default
Changed default test databases from ckan_test and ckan_test_datastore to
ckan_test and datastore_test
Changed default ckan.site_id from ckan.net to default
Disabled file log handler, rely on Apache's log files only (as we do on our
servers)
Changed the docs to use restructured text substitutions for things like the
paths to the virtualenv and config files, database names, etc. instead of
typing them out each time
Changed default FileStore storage dir from /my/path/to/storage/root/directory to /var/lib/ckan/default
Lots of minor formatting fixes and editing

So now we have:

virtualenv:        /usr/lib/ckan/default
development.ini:   /etc/ckan/default/development.ini
production.ini:    /etc/ckan/default/production.ini
ckan.site_id:      default
Database user:     ckan_default
Database:          ckan_default
Datastore user:    datastore_default
Datastore db:      datastore_default
Test database:     ckan_test
Test datastore db: datastore_test
Filestore dir:     /var/lib/ckan/default

Update the source install and related docs (datastore setup, solr setup, testing) and files (deployment.ini_tmp, test-core.ini) for CKAN 2.0. The idea is to document the setup process that OKFN is currently using on its servers, so that the deployment docs can become the docs for the OKFN dev team and we can dogfood our own docs, instead of having a bunch of wiki pages, READMEs, etc. that we follow and ignoring the "official" docs. Hopefully those can all be deleted or cut down massively once this is done. So both the docs now describe the layout we use on our servers, where there's space for multiple CKAN sites on the same server, with virtualenvs in /usr/lib/ckan/ and config files in /etc/ckan/. - Changed default virtualenv location from `~/pyenv` to `/usr/lib/ckan/default` - Changed default config files dir to `/etc/ckan/default` - Added tip for developers in source install docsS symlink `/usr/lib/ckan` and `/etc/ckan` to directories in your homedir to install ckan in your homedir - Changed default database user from `ckanuser` to `ckan_default` - Changed default database from `ckan_dev` to `ckan_default` - Changed default datastore database from `datastore` to `datastore_default` - Changed default datastore database user from `readonlyuser` to `datastore_default` - Changed default test databases from `ckan_test` and `ckan_test_datastore` to `ckan_test` and `datastore_test` - Changed default `ckan.site_id` from `ckan.net` to `default` - Disabled file log handler, rely on Apache's log files only (as we do on our servers) - Changed the docs to use restructured text substitutions for things like the paths to the virtualenv and config files, database names, etc. instead of typing them out each time - Lots of minor formatting fixes and editing So now we have: virtualenv: /usr/lib/ckan/default development.ini: /etc/ckan/default/development.ini production.ini: /etc/ckan/default/production.ini ckan.site_id: default Database user: ckan_default Database: ckan_default Datastore user: datastore_default Datastore db: datastore_default Test database: ckan_test Test datastore db: datastore_test

Make them fit with the changes in commit 6e1d325

Uses the same directory structure as the updated source install docs, so you can step through the source install docs and then the deployment docs as written and it all works. Also updated the apache.wsgi and sites-available files to match those that we use on our servers. Removed CORS setup instructions as this has been builtin since CKAN 1.5.

Minor edits to install from source docs

Minor edits to solr setup docs

Update the database and user names in travis build

Minor edits to the datastore docs, from Dominik. See: #807

nigelbabu · 2013-05-06T10:24:21Z

doc/conf.py

+.. |sstore| replace:: |config_dir|/sstore
+.. |storage_parent_dir| replace:: /var/lib/ckan
+.. |storage_dir| replace:: |storage_parent_dir|/default
+.. |restart_apache| replace:: sudo service apache2 reload


This should either be called reload_apache or the command should be sudo service apache2 restart.

I'll change it to reload. Let's use reload everywhere, and only put a restart if we find somewhere that it's actually needed. (I'm not sure what the difference is anyway.)

We've had reports from some users that this causes problems with some CKAN features, so we don't want to recommend it until tested.

seanh · 2013-05-06T12:41:43Z

@nigelbabu Ok, I've made the changes you suggested. Only thing is the question about copying development.ini or running paster make-config again, not sure what to do there but I'm happy to go with your recommendation if you have one

joetsoi · 2013-05-06T14:53:34Z

Eitherway they'll both end up being generated from deployment.ini_tmpl. http://docs.pylonsproject.org/projects/pylons-webframework/en/latest/deployment.html#make-config it doesn't look like there was a way of specifying a different template file

paster make-config is generated at this point for the packaging in a post install script so everyone gets a different beaker.session_secret as users are most likely too lazy to change it. (We should do the same for the repoz.who stuff that was mentioned in the standup on friday as well)

The package currently also sets up nginx, it might be a good idea to just have it setup on apache as it's only there because that's how the data.gov stuff is deployed. Also 8080 is the default port for a bunch of stuff (e.g tomcat). If something's running on 8080 then the package will break and I'm worried we'll generate a whole bunch of support requests of "Hey the package install is broken!" where we end up figuring out that port is in use.

nigelbabu · 2013-05-07T02:27:48Z

doc/install-from-source.rst

+
+   .. parsed-literal::
+
+       pip install -e 'git+\ |git_url|\#egg=ckan'


This is a little tricky. We can't recommend this if someone wants to install CKAN for development.

nigelbabu · 2013-05-07T02:32:39Z

We use Nginx for our deployments as well, so if we're going to have unified instructions, having it listed is a good idea.

nigelbabu · 2013-05-07T02:34:52Z

the apache.wsgi file is in the package source. Once this is merged, I'll make sure the package stuff is same as what the source install recommends.

seanh · 2013-05-07T08:29:09Z

@nigelbabu Ok:

I don't see why a development install can't use pip install -e.
For nginx, I agree it would be good to add this but I don't know how we set it up (we've never had docs for this afaik)
Do you think the apache.wsgi file could go into git, and then be imported by both the docs and the packaging? (We could leave this for 2.1)

seanh · 2013-05-07T09:32:36Z

I double-checked (since I don't use it myself) pip install -e is fine for dev installs
I think it would be good to put nginx into the deployment docs, but can this wait for 2.1? It should have a brief (one sentence) explanation of why you would want to use nginx and apache together for CKAN, and then how to do it. @nigelbabu Can you make a ticket for this in 2.1? And do you also want to assign it to yourself? :)
apache.wsgi file let's get this sorted for 2.1

Are there any other issues?

nigelbabu · 2013-05-07T11:55:59Z

Sounds good!
2 and 3. I'm happy for us to finalize those in 2.1. I opened issue Add documentation for Nginx configuration with CKAN #877 for that.

Merging this in now.

519 update deployment docs second try

seanh · 2013-05-07T14:09:34Z

Cherry-picked into release-v2.0

Sean Hammond added 6 commits May 5, 2013 14:44

[#519] Update post-installation docs

02f5bad

Make them fit with the changes in commit 6e1d325

[#519] Edit source install docs

02cf77c

Minor edits to install from source docs

[#519] Edit solr setup docs

d70fc0b

Minor edits to solr setup docs

[#519] Update FileStore setup instructions

c634a67

seanh mentioned this pull request May 5, 2013

[wip] [#519] Update deployment docs for CKAN 2.0 #821

Closed

13 tasks

ghost assigned nigelbabu and seanh May 5, 2013

Sean Hammond added 2 commits May 5, 2013 18:43

[#519] Fix Travis build

9197182

Update the database and user names in travis build

[#519] Minor edits to solr setup docs

a64c21e

seanh mentioned this pull request May 6, 2013

Minor tweaks in Solr docs #814

Closed

[#519] Add postfix to deployment docs

427ee0f

seanh mentioned this pull request May 6, 2013

error using curl test after following datastore setup instructions #807

Closed

[#519] Minor edits to datastore docs

6cbdcb1

Minor edits to the datastore docs, from Dominik. See: #807

ghost assigned nigelbabu May 6, 2013

nigelbabu reviewed May 6, 2013
View reviewed changes

Sean Hammond added 3 commits May 6, 2013 14:32

[#519] Docs: change restart_apache substitution to reload_apache

7ced6fc

[#519] Docs: use the reload_apache substitution

6032a29

[#519] Remove non-root URL deployment instructions

1240f02

We've had reports from some users that this causes problems with some CKAN features, so we don't want to recommend it until tested.

nigelbabu reviewed May 7, 2013
View reviewed changes

Sean Hammond added 2 commits May 7, 2013 10:23

[#519] Replace seanh with whoami in source install docs

b6f7878

[#519] Docs: changed 'supported' to 'recommended'

be59aaa

nigelbabu added a commit that referenced this pull request May 7, 2013

Merge pull request #874 from okfn/519-update-deployment-docs-second-try

e88b2fb

519 update deployment docs second try

nigelbabu merged commit e88b2fb into master May 7, 2013

nigelbabu deleted the 519-update-deployment-docs-second-try branch May 7, 2013 11:56

timrdf mentioned this pull request May 20, 2013

add CKAN install to install.sh timrdf/prizms#3

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

519 update deployment docs second try #874

519 update deployment docs second try #874

seanh commented May 5, 2013

nigelbabu May 6, 2013

seanh May 6, 2013

seanh commented May 6, 2013

joetsoi commented May 6, 2013

nigelbabu May 7, 2013

seanh May 7, 2013

nigelbabu commented May 7, 2013

nigelbabu commented May 7, 2013

seanh commented May 7, 2013

seanh commented May 7, 2013

nigelbabu commented May 7, 2013

seanh commented May 7, 2013


		.. parsed-literal::

		pip install -e 'git+\ \|git_url\|\#egg=ckan'

519 update deployment docs second try #874

519 update deployment docs second try #874

Conversation

seanh commented May 5, 2013

nigelbabu May 6, 2013

Choose a reason for hiding this comment

seanh May 6, 2013

Choose a reason for hiding this comment

seanh commented May 6, 2013

joetsoi commented May 6, 2013

nigelbabu May 7, 2013

Choose a reason for hiding this comment

seanh May 7, 2013

Choose a reason for hiding this comment

nigelbabu commented May 7, 2013

nigelbabu commented May 7, 2013

seanh commented May 7, 2013

seanh commented May 7, 2013

nigelbabu commented May 7, 2013

seanh commented May 7, 2013