Search

[cfra]

Code Version

1.1.5

Expected Behavior

Looking at thiss-js example configs, it seems like pyFF should provide a search functionality at the following url:
http://pyff/api/search?q=needle.

Current Behavior

Instead, a 500 error is generated instead:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/lib/encoding.py", line 223, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 388, in default
    return self.server.request(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 598, in request
    r = p.process(self.md, state=state)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 302, in process
    return Plumbing.Request(self, md,
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 253, in process
    return pl.iprocess(self)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 356, in when
    return Plumbing(pipeline=req.args, pid="%s.when" % req.plumbing.id).iprocess(req)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 324, in _pipe
    ot = Plumbing(pipeline=req.args, pid="%s.pipe" % req.plumbing.id).iprocess(req)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 355, in when
    if c is not None and (not values or _any(values, c)):
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 250, in _any
    if x in d:
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 432, in __getitem__
    return cptools.accept(item, debug=True)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/lib/cptools.py", line 570, in accept
    if element.value in media:
TypeError: argument of type 'int' is not iterable

Steps to Reproduce

Not sure what information I can provide. I am using the following pipeline configuration, maybe I made a mistake there:

- when update:
    - load:
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-test-metadata.xml dfn-aai.pem"
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-basic-metadata.xml dfn-aai.pem"
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-edugain+idp-metadata.xml dfn-aai.pem"
    - break
- when request:
    - select
    - pipe:
        - when accept application/xml:
             - xslt:
                 stylesheet: tidy.xsl
             - first
             - finalize:
                cacheDuration: PT10D
                validUntil: PT5H
             - sign:
                 key: default.key
                 cert: default.crt
             - emit application/xml
             - break
        - when accept application/json:
             - xslt:
                 stylesheet: discojson.xsl
             - emit application/json:
             - break

[leifj]

Can you try it with the discojson pipeline (builtin) instead of using a stylesheet?

[cfra]

Sorry if this is a stupid question, but how can I make the docker image run the discojson pipeline?

[leifj]

You replace the - xslt: stylesheet: discojson.xsl with just - discojson in that pipeline

[cfra]

I have updated the pipeline to look like this:

- when update:
    - load:
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-test-metadata.xml dfn-aai.pem"
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-basic-metadata.xml dfn-aai.pem"
        - "http://www.aai.dfn.de/fileadmin/metadata/dfn-aai-edugain+idp-metadata.xml dfn-aai.pem"
    - break
- when request:
    - select
    - pipe:
        - when accept application/xml:
             - xslt:
                 stylesheet: tidy.xsl
             - first
             - finalize:
                cacheDuration: PT10D
                validUntil: PT5H
             - sign:
                 key: default.key
                 cert: default.crt
             - emit application/xml
             - break
        - when accept application/json:
             - discojson
             - emit application/json:
             - break

The error for a get request with accept application/json to http://pyff/api/search?q=Test still looks similar to me:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cprequest.py", line 638, in respond
    self._do_respond(path_info)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cprequest.py", line 697, in _do_respond
    response.body = self.handler()
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/lib/encoding.py", line 223, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/_cpdispatch.py", line 54, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 388, in default
    return self.server.request(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 598, in request
    r = p.process(self.md, state=state)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 302, in process
    return Plumbing.Request(self, md,
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 253, in process
    return pl.iprocess(self)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 356, in when
    return Plumbing(pipeline=req.args, pid="%s.when" % req.plumbing.id).iprocess(req)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 324, in _pipe
    ot = Plumbing(pipeline=req.args, pid="%s.pipe" % req.plumbing.id).iprocess(req)
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 281, in iprocess
    raise ex
  File "/usr/local/lib/python3.9/dist-packages/pyff/pipes.py", line 271, in iprocess
    ot = pipefn(req, *opts)
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 355, in when
    if c is not None and (not values or _any(values, c)):
  File "/usr/local/lib/python3.9/dist-packages/pyff/builtins.py", line 250, in _any
    if x in d:
  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 432, in __getitem__
    return cptools.accept(item, debug=True)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/lib/cptools.py", line 570, in accept
    if element.value in media:
TypeError: argument of type 'int' is not iterable

[leifj]

what client are you using here? curl? Can you send me the cmdline you use? Also - are you using docker, which docker-container are you using ?

[cfra]

While I was using the Firefox extension Rester earlier, I can reproduce the issue with curl and the following command line:
curl -H "Accept: application/json" 'http://pyff/api/search?q=Test'

I am running the docker image docker.sunet.se/pyff:stable@sha256:21acef59fa2fe2f625894fb100f7a69b5d33d8cb0f6c62bbfa018c57b3911093.

[leifj]

On 2021-11-09 17:47, Christian Franke wrote: While I was using the Firefox extension Rester earlier, I can reproduce the issue with curl and the following command line: |curl -H "Accept: application/json" http://pyff/api/search?q=Test| I am running the docker image ***@***.***:21acef59fa2fe2f625894fb100f7a69b5d33d8cb0f6c62bbfa018c57b3911093|.

Can you try to build that docker image yourself from github.com/sunet/docker-pyff - I'm not sure if we maintain the stable tag.

[fredrikdomeij]

I have built the docker image myself. I still get the same error:

  File "/usr/local/lib/python3.9/dist-packages/pyff/mdx.py", line 432, in __getitem__
    return cptools.accept(item, debug=True)
  File "/usr/local/lib/python3.9/dist-packages/cherrypy/lib/cptools.py", line 570, in accept
    if element.value in media:
TypeError: argument of type 'int' is not iterable

This does though seem to have to do with the Accept header handling, the following pipeline does not have this problem:

- when update:
    - load:
        - http://mds.edugain.org
    - break
- when request:
    - select
    - pipe:
      - discojson
      - emit application/json:

It just always returns json.
I do still have problem with the /api/search?q=xxx url that is claimed to work (see SEARCH_URL at https://pyff.readthedocs.io/en/latest/usage/deploying.html). What is working however is /search?query=xxx, but thiss-js is using q=xxx as query parameter, not query=xxx.
Trying to read the pyFF code I see

src/pyff/api.py:
    match = request.params.get('q', request.params.get('query', None))

But I cannot tell how that affects the q/query issue above.

I suggest there is some bug regarding the "when accept" handling, and that "q" should be an alias for "query" (or the other way around).

[fredrikdomeij]

Works when running pyFF 2.0.0!