"Running a server" documentation outdated

[dirkdevriendt]

Thanks for the hard work on this project!

This problem may be due to my lack of experience with Python Paste (and the fact the documentation on http://pythonpaste.org/ has been down for two months now), but we are trying to setup a pydap docker instance and have trouble getting the instructions to work for us.

(env) root@cf53f30390c0:/var/www/pydap# paster create -t pydap server
Traceback (most recent call last):
  File "/var/www/pydap/env/bin/paster", line 11, in <module>
    sys.exit(run())
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 102, in run
    invoke(command, command_name, options, args[1:])
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 141, in invoke
    exit_code = runner.run(args)
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/command.py", line 236, in run
    result = self.command()
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/create_distro.py", line 72, in command
    self.extend_templates(templates, tmpl_name)
  File "/var/www/pydap/env/lib/python2.7/site-packages/paste/script/create_distro.py", line 264, in extend_templates
    'Template by name %r not found' % tmpl_name)
LookupError: Template by name 'pydap' not found

Is there a way to get some guidance on what is needed to get paste to work?

Even better might be documentation or a code update to get a uwsgi server setup without having to pass via Python Paste? There is no conventional "application" method in the wsgi/app.py and the "main" method relies on gunicorn.app.pasterapp.paste_server as well.

Paste seems to add significant complexity for a limited benefit

[jameshiebert]

Hi @dirkdevriendt! Thanks for the heads up on this. You're right; the docs are probably out of date in this section... I know that I personally haven't tried to use Paster for this for years (my organization runs Pydap using Flask, gunicorn and supervisord.

We'll leave this issue open until someone gets to writing up the docs. In the mean time, I can advise that you can use pydap.wsgi.app:DapServer class, initialized with the path to your data files (a DapServer object is a WSGI callable). Then you can expose that as your "app" to any WSGI framework.

It would looks something (very rough approximation) like this:

from flask import Flask
from pydap.wsgi.app import DapServer

pydap_inst = DapServer('/path/to/my/data/files')
app = Flask(__name__)
app.wsgi_app = pydap_inst
app.run('0.0.0.0', 8000)

YMMV.

[dirkdevriendt]

Excellent info!
I have created a docker image, built with nginx + uWSGI + Flask (based on https://hub.docker.com/r/tiangolo/uwsgi-nginx-flask/)

We'll be testing a bit and if it works properly, I can send you the resulting setup so you can create a docker repo from it if you want.

[dirkdevriendt]

Seems to work just fine (note: we're pulling the netcdf handler from github since the pypi version is not up-to-date)

I'll add a stripped down setup here so you can see what it looks like, but if you're interested you probable want to either integrate the relevant files in the pydap repo or create a dedicated docker repo. I can help set it up when you decide if/how you want to proceed.

./README.md

build docker image:
`docker build . -t "docker-server-opendap"`

start docker instance:
`docker run -p 80:80 -v /data:/data docker-server-opendap`

environment variables:
  - APP_DEBUG (default: False)
  - APP_PATH (default: /data)
  - APP_HOST (default: 0.0.0.0)
  - APP_PORT (default: 80)

./Dockerfile

FROM tiangolo/uwsgi-nginx-flask:flask

COPY ./requirements.txt .
COPY ./app /app
COPY ./pydap.conf /etc/nginx/conf.d/

RUN \
 apt-get update && \
 apt-get install -y netcdf-bin libnetcdf-dev nano rsync && \
 pip install --upgrade pip && \
 pip install -r requirements.txt

RUN rsync -r /usr/local/lib/python2.7/site-packages/pydap/wsgi/templates/static/ /app/static/

./pydap.conf

server {
    client_max_body_size 10M;
    location / {
        try_files $uri @app;
    }
    location @app {
        include uwsgi_params;
        uwsgi_pass unix:///tmp/uwsgi.sock;
    }
    location /static {
        alias /app/static;
    }
}

./requirements.txt

netCDF4
Pydap[functions,cas]==3.2.0
git+https://github.com/pydap/pydap.handlers.netcdf.git

./app/main.py

import os
from flask import Flask
from pydap.wsgi.app import DapServer

app_debug = (os.environ.get('APP_DEBUG', 'False').lower() in ['true', 'yes', 'y', '1'])
app_path = os.environ.get('APP_PATH', '/data')
app_host = os.environ.get('APP_HOST', '0.0.0.0')
app_port = int(os.environ.get('APP_PORT', '80'))

app = Flask(__name__)
app.wsgi_app = DapServer(app_path)

if __name__ == "__main__":
   app.run(host=app_host, debug=app_debug, port=app_port)

[jameshiebert]

Cool, thanks for putting that together! I think that it would probably be best to (as the title of this issue indicates) update the documentation with the information, and not necessarily integrate it directly into the main repo. I would see it as "guidance on how to use PyDAP in the real world" and not necessarily as executable code that will 100% be up-to-date at all times (especially given the limited volunteer bandwidth behind PyDAP at present). If we did add them to the repo, I would probably want to put them in a "contrib" dir, and temper expectations regarding its maintenance.

But I'd be happy to see a PR that integrates the above into a section in the docs (maybe replacing the Paster section).

[dirkdevriendt]

I have no intention to change your mind or to push more work onto the pydap volunteers, but putting this in documentation seems counterproductive since it makes the setup (appear) harder, which is not what docker is about :-)
The thing is that with the code in a repo + an automated build on the docker hub, the quickstart + server page can be replaced with:

1. install docker if you don't have it yet
2. execute 'docker run' command
note: check [link to repo] for details

If you prefer to not add it under the pydap organisation, no problem, I'll see if I can set something up myself or propose a PR to an existing repo (https://github.com/cynici/pydap from @cynici, maybe?)

[jameshiebert]

since it makes the setup (appear) harder

Yeah, that does make sense. I'll give it some thought and others are free to chime in to the discussion :)

[marty-sullivan]

My two cents, it would be better to have something in the docs that actually works which apparently the paste script no longer does ;)

[tomkralidis]

+1 agree. I just tried installing pydap from scratch via the Quickstart and run into this error as well.

[marty-sullivan]

I suppose it might even make more sense to point people to Hyrax instead of Pydap for new server deployments.

[ghost]

@jameshiebert I tried your code ,but I get error!!

Error { code = -1; message = "Traceback (most recent call last): File "/usr/local/lib/python2.7/site-packages/pydap/handlers/lib.py", line 93, in __call__ dataset = self.parse(projection, selection, buffer_size) File "/usr/local/lib/python2.7/site-packages/pydap/handlers/lib.py", line 120, in parse if self.dataset is None: AttributeError: 'Handler' object has no attribute 'dataset' "; }