Up-to-dated view server

[djc]

From kxepal on August 16, 2010 16:19:48

This is port of javascript view server for couchdb server 0.10+
Support of shows, lists, filters and validate_doc_updates for pythonic views included.

Due to currently there is no couchdb-python API to manipulate this functions n, I didn't wrote any tests, but it passed all tutorial examples.

Original issue: http://code.google.com/p/couchdb-python/issues/detail?id=146

[djc]

From djc.ochtman on August 16, 2010 07:37:00

Did you write this yourself? Is it a port of the JavaScript view server, or did you find some documentation on the view protocol?

I definitely want an updated view server, but I prefer that we only add something for which we have good tests, so that we can prevent it from going stale again. Would you be in a position to write tests for this? Some design notes on the protocol would likely also be helpful.

If we decide to take this, I also have some code/style nits.

Labels: -Type-Defect Type-Enhancement

[djc]

From kxepal on August 16, 2010 07:59:03

It's javascript view server ported by me. I've going by only one thing - to make view server maximum compatible with js one, because it's always up-to-dated and have released first. So as close pythonic view server will be to it so easy support will be.

Shame on me, I've thought for some unknown reason that view server must be tested via couchdb-python API - just have look at tests/view.py. I'll make them for tomorrow.

[djc]

From kxepal on August 19, 2010 06:38:14

I've found a little problems due passing official view server tests, but now all fine.

I'll make better exceptions handling + crush tests later - I really dont like this forest of try..except, but couldnt invent something better now - need some time to think about it. So this is just check point(:

In additional to javascript view-server features I've implemented two behaviors:

• sealed document: changing document in map function makes no sense for other map functions within single view.
• any pythonic exception will crush view server while javascript view server allow crushing only on fatal errors - I dont know which ones python have.
  Everything else seems same.

[djc]

From kxepal on August 19, 2010 06:42:57

oops, forgot to remove debug handler from view server.

[djc]

From kxepal on September 03, 2010 04:31:24

Ok, here is updated view server. Major changes:

• support any couchdb server since 0.9 version. By default view server works in mode of compatibility with latest couchdb server version, just run it with --couchdb-version key, e.g. view.py --couchdb-version=0.10, to make it work well for 0.10 couchdb server.
• Tests are included for each supported version - all of them had been ported from javascript view server.
• Assertion error within validate_doc_update function doesn't count as Fatal like any other pythonic exception and will be wrapped as Forbidden error
• "Reduce output must shirnk more rapidly" error now may be occured
• More verbose debug logging

Also I have to add function versionizing decorator to split their behavior for each couchdb version. I thinks it will be useful in future to keep legacy support without serious code rewriting.

[djc]

From kxepal on September 13, 2010 05:09:33

Changes:

• fixed compatibility for python-2.4+
• added more verbose debug output

I've removed old attachments, because they are not actual for now.

Attachment: view.py util.py test.py testutil.py

[djc]

From djc.ochtman on September 19, 2010 06:16:15

I'm going to need to find a solid chunk of time to review this, which might take a while, but I'd really like to take this for the next release...

[djc]

From djc.ochtman on December 22, 2010 01:13:46

Issue 140 has been merged into this issue.

[djc]

From kxepal on February 12, 2011 11:08:27

I've fixed issue #163 in this view server https://code.google.com/r/kxepal-couchdb-python-featured/source/detail?r=e1448890d2223e0321f96459a53c1757dc5b9662 (just have not seen any reasons to attach once again all files for several small changes)

In over way it's ready for 1.0.2 since main change in js view server was in sealing documents for map func, but this feature is done already.

I will port sofa and tapirwiki for this view server within next two weeks, so this made a great challenge for him and may be more fixed will come if I've missed something.

[djc]

From kxepal on March 22, 2011 04:06:28

I've found one thing missed - require function to have some modular application within document design. But there are some questions about it:

• should it work as in javascript view-server: wrap some abstract code and return export values? Would be better to implement python-like import?
• should it support eggs? I think it should, but I have no idea how to import eggs inline without saving them on disk. This could be a problem for application hosters.
  any other behavior suggested?

[djc]

From extempor...@gmail.com on March 22, 2011 08:30:54

What I'm doing to handle some efficiency issues with import:

def fun():
import datetime
... other imports ...
def fun(doc):
...
delta = (datetime.strptime(doc[...]) - datetime.strptime(doc[...])).days
...
return fun
fun = fun()

What really ought to happen is that the view server should go through each variable in the exec's locals and check the module. If module exists, but is None, and the variable points to a callable, then use that as the map/reduce func, and error out if more than one is found. This would be backwards compatible with existing view functions, but would make it so closures are not necessary.

I think the eggs deployment issue has been tackled many times before. Importing eggs inline would obliterate responsiveness. Does couch time you out if you take to long? I'd be concerned that it would/should. If an application hoster supports python, then sooner or later they'd need to come up with a solution to handle 3rd party software since, frankly, the power of using python as a view server doesn't just lie in "it looks nice" and "it has yield".

To meet couch's same-code-same-result requirements (no side effects), we would maybe have to mark imports with python and module version strings, and push that back into couch. For example, 'import mythirdpartyegg' would then append '#mythirdpartegg py cpython-2.6.5 mod 3.11r7112' to the end of the eval string, one per unique detected module. Any time you do any module upgrades, you can just delete the version-marking comments out of the view func manually, and couch will regenerate it.

However, most modules tend to be stable enough API-wise that this isn't a problem. If there were any behavior altering bugs/changes to the code, an administrator could achieve this manually.

[djc]

From kxepal on March 23, 2011 13:45:20

Several questions, could I?

1. what the reason of such wrapper against:
   def fun(doc):
   ...
   delta = (datetime.strptime(doc[...]) - datetime.strptime(doc[...])).days
   ...
   import datetime

this is not very pythonic to place imports below, but datetime will be tried to import only once. However, this style could produce another problem: views must have not any state and any dependence from source which could be changed later.

2. Hoster could provide 3rd party modules, but it couldn't provide all versions of each template engine, for example. May be you needs trunk jinja2 with you own patches, who knows? So idea to create fully portable pythonic couchapp will be failed.

3. Is preprocessors statements really good idea? I saw them in couchapp, but they have been used only for declaration, not within document design nor view server.

As intermediate result, this is implementation of require function as is it works for javascript view server: https://code.google.com/r/kxepal-couchdb-python-featured/source/detail?r=0b6625db473e74a83df7e9a339899a3c318f7b80 I still need to finish some details, so attachments with new version of view server will be later. Sorry, Dirkjan, it seems you to have revise it once again, but I'll include documentation for each vital function and more tests to make process more easy just(:

[djc]

From extempor...@gmail.com on March 24, 2011 11:27:04

You could put imports after the inner func that needs them, and python's scoping rules would resolve variable lookups, but I agree, it's not Pythonic. Closures themselves are not very Pythonic, either. Anyways, why not put imports first, as per my suggestion?

The whole reason for using a closure is to avoid the performance penalty of repeating the imports for each document. If a function needed k imports and there are n documents in a couch database, then, compared to the closure technique shown above, there'd be n*k redundant imports taking place, which is very slow (python doesn't re-import the module, but there is overhead involved, which can be significant).

See: http://wiki.python.org/moin/PythonSpeed/PerformanceTips#Import_Statement_Overhead I disagree with you on the severity of Couch's "no side-effects" requirement for view/show/list functions. It's a matter of practicality, not a matter of theory. Yes, Couch says that the same document passed to the same code must produce the same output, no matter how many times it's executed. If the document doesn't change, then the result doesn't change.

However, this mandate is only for data correctness. If the module you're importing in a view function gets upgraded, and its behavior changes, but your view function stays the same (so couch doesn't regenerate the view), then all that happens is that your view will be incorrect. Couch won't break (couch won't know, mind, or even care).

Also, using module imports don't count as "side effects" at all. Aside from the random module, almost all modules (including 3rd party ones) are stateless in their behavior. For example, it'd be fine to describe a shape as a list of points in a couch document, and then use PIL to draw that to a png image in a show function, or to use couch to store server access logs, and then use pychart from a list function to generate an svg rendered line graph of server traffic.

Furthermore, the above "same code, same doc, same result" requirement does not apply across all time. For example, I could define a view function that imported some module, and then change the behavior of the module. All I'd have to do fix the consistency of the view would be to add a single space to the end of the view function, save the design document to couch, and then remove that space, and save the design document again. The code is exactly the same as before, yet we side-stepped the upgrade-changes-behavior issue, and caused couch to regenerate its views to reflect the new behavior. And this process can be automated (alternately, you could delete the views from the design doc, do a view cleanup, and then reupload the original design document, which would require only one view regeneration instead of two). As long as you regenerate your views when needed, the "could be changed later" issue isn't a problem at all.

In general, hosting vendors are not ever going to support all versions of all potential packages. They're either going to support only a handful of popular modules (probably Django templating) and force long release cycles, or they'll provide you with a few megabytes of space to upload your own modules in your own private module path, or they'll not allow the use of any kind of 3rd party modules (in which case you might as well use Javascript).

[djc]

From kxepal on March 24, 2011 12:40:47

At first till I don't forget, thanks you for detail reply(:

So about imports:
Allow to have them on top of design function as PEP told us I see is ok too and this is much more intuitive behavior.
However, only map functions are cached: reduces/shows/lists/updates and others are recompiling for each call, so all this import optimization tricks are not so useful as they have to be.
I suppose that also would be better to extend preimported packages with most popular and useful, which probably would be always imported, such as:
time, datetime, re, hashlib, math, random, itertools and others. But that would be very implicitly feature without reading of docs and not only I should decide what will be in this list.

No side effect is requirement for views only, afaik, because index is based on view result only, while shows/lists just the way to show data in nicer form.
There is one more thing to keep views as much stable and independent from side effects as possible: if you have dozen millions documents last thing that you would like to do is to rebuild view index, because this would take hours. Yes, trick with secondary server and replacement view index is nice idea, but you still have to lose your hours and you'll have service down for a some time.
However, in 1.1.x branch was added feature to require view/lib stored module for map functions.

Suddenly for couchdb view servers, hoster wouldn't provide some space for your own modules because that would require some additional interface, monitor to reload module set in realtime without forced restart of view server and...this solution killing portable pythonic couchapps. Javascript couchapps are awesome because you just have to type: "couchapp push" and that's all - it works!

So, what resolution will be?

[djc]

From extempor...@gmail.com on March 25, 2011 23:04:25

Can you link some documentation on that 1.1.x feature? That's something I'd be very interested in learning about.

Show and list functions are supposed to be side effect free too. That way, they can be cached by couch (though I'm unsure if couch itself actually does that). I'm pretty sure couch does proper Etag handling of the show/list results, so if you're expecting that you can generate a new result each time someone accesses a doc via show, or a set of documents via list, know that couch will tell the client/browser to use what it already has if none of the pertinent documents in the database have changed.

Check out:

• http://guide.couchdb.org/draft/show.html#constraints * http://guide.couchdb.org/draft/transforming.html#example (see the first "lightbulb" blockquote in that section).

Hah, you're right about the map caching. I forgot that some of the others don't cache! Hmmmmm. We could do our own caching. I'm not sure if that's considered bad behavior or not, but I don't see how it makes a difference, and really, I think the fact that they send the reduce function every time a reduce computation is needed is a bad choice in protocol design -- it's simpler, yes, but they could have just added 'load' and 'unload' commands for functions, so that you can do one-time compilation.

What we can do is cache the reduce/list/show functions they give us and run the computation. Next time they pass us a function, we do a string compare on the new code for that function to the string of code we originally received for that named list/show/reduce function. If it's the same as before, then our compilation step becomes a no-op, and we just use what we already had. If the function is different, then we assume that the design doc has been updated, and we recompile. This way, we can do things like use closures for those performance gains. Couch's own rules and reasons for side-effect free functions are what gives us the right to do this kind of caching.

Moving on... well random probably shouldn't be imported (or at least not used by any couch stuff), since by its very nature, it'll produce different results every time.

As for downtime, in many cases, couch can service requests while an index is being rebuilt. Also, you can easily replicate to another database/server/whatever (secondary server), rebuild it there, and temporarily make that the primary database your serving from while you rebuild the index on your real primary. That sounds complicated, but as we all know, in couch that takes less thought than it took me to write about it just now. Also, that's all assuming that couch's index hot-rebuild doesn't cover your use case. Hot-swapping couch databases and even couch server instances, or adding redundancies and failovers is a fact of life with couch -- sure, there are plenty of us running just one couch instance for a given application, but it's so painless to temporarily add another copy ad-hoc, and tear it back down when you don't need to again. Unlike with other systems, you don't even really need to plan ahead when you do it.

You've got a good point. I'm not sure what the resolution would be. Clearly python wins over javascript for couchdb not because of its pretty and concise syntax, since view/list/show functions are about the same size in either language if you aren't allowed to import anything. Python would win out because of its standard library (which is API-stable enough for couch), and because of its 3rd party modules.

In any case, behavior-changing module upgrades could only be handled by rebuilding the index. Even though the code that couch can see (the code stored in the design doc) hasn't changed, the code it links to has. So you simply have to treat it in same way as if you changed a line of code in the map func itself, and there's no way around that.

Just like with retooling your own view/list/show function code, you have to strike a balance between the time you need to spend rebuilding an index, and the benefits you get from changing the code. After all, you can always choose to not upgrade your python or module to a new version, and just because there is a new version, doesn't mean you need it.