Adds a RobotsTag middleware

[rik]

and a bit of cleanup at the same time

[jsocol]

Indent needs to line up inside the open paren, or these should stay on separate lines.

[jsocol]

You should look at commonware.response.decorators and look at the xframe_* decorators. I think we should copy at least two concepts here. One is the @xrobots_exempt() decorator, which I alluded to in the inline comment. The other is a kinder way to let people set the X-Robots-Tag value.

There are very few values for X-Frame-Options, so it made sense to just create 3 simple decorators. There are many, many values for X-Robots-Tag, so I think in this case you want a slightly more complex decorator:

def xrobots_tag(rule='nofollow'):
    def _decorator(f):
        @wraps(f)
        def _wrapped(req, *a, **kw):
            res = f(req, *a, **kw)
            res['x-robots-tag'] = rule
            return res
         return _wrapped
    return _decorator

Then people can more easily do

@xrobots_tag('noindex,nofollow')
def myview(request):
    pass

[jsocol]

NB, I am also asking for an @xrobots_exempt decorator but you should basically be able to copypasta @xframe_allow for that one.

[jsocol]

Rebased, added decorators and a pile of tests.

6147e97...f95a03a

[rik]

Thanks for filling in!