Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Something brokes SearXNG search api when i choose json format and videos category #2505

Closed
mbaroncini opened this issue Jun 17, 2023 · 2 comments · Fixed by #2508
Closed

Something brokes SearXNG search api when i choose json format and videos category #2505

mbaroncini opened this issue Jun 17, 2023 · 2 comments · Fixed by #2508
Labels
bug Something isn't working

Comments

@mbaroncini
Copy link

mbaroncini commented Jun 17, 2023 •
edited
Loading

Version:
2023.6.16+71b6ff07
docker image: searxng/searxng:latest

What happened?
When i try http://localhost:8080/search?q=test&format=csv&categories=videos this works both on chromium broswer and via terminal using curl :curl "http://localhost:8080/search?q=test&format=csv&categories=videos"

This instead works only on browser: http://localhost:8080/search?q=test&format=json&categories=videos, when i try on terminal curl "http://localhost:8080/search?q=test&format=json&categories=videos" i obtain

<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

How To Reproduce
Run searxng with docker:

mkdir test
docker pull searxng/searxng
docker run -d --rm \
              -d -p 8080:8080 \
              --name searxng_test \
              -v "./test:/etc/searxng" \
              -e "BASE_URL=http://localhost:8080/" \
              searxng/searxng

update formats on test/settings.yml

  # formats: [html, csv, json, rss]
  formats:
    - html
    - json
    - csv

restart the docker container:

docker restart searxng_test

Now if you try to query the search api with curl "http://localhost:8080/search?q=test&categories=videos&format=csv" should be fine.

Instead : curl "http://localhost:8080/search?q=test&categories=videos&format=json" raises an error 500

Expected behavior
The json format should work as the csv does.

Screenshots & Logs
These are the docker container logs:


searxng_  | DEBUG   searx.network.vimeo           : HTTP Request: GET https://vimeo.com//search/page:1?q=test "HTTP/1.1 200 OK" (text/html; charset=UTF-8)
searxng_  | DEBUG   searx.webapp                  : http%3A//localhost%3A8080/search%3Fq%3Dtest%26format%3Djson%26categories%3Dvideos uses locale `en`
searxng_  | ERROR   searx.webapp                  : Exception on /search [GET]
searxng_  | Traceback (most recent call last):
searxng_  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 2190, in wsgi_app
searxng_  |     response = self.full_dispatch_request()
searxng_  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1486, in full_dispatch_request
searxng_  |     rv = self.handle_user_exception(e)
searxng_  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1484, in full_dispatch_request
searxng_  |     rv = self.dispatch_request()
searxng_  |          ^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/site-packages/flask/app.py", line 1469, in dispatch_request
searxng_  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
searxng_  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/local/searxng/searx/webapp.py", line 764, in search
searxng_  |     response = json.dumps(x, default=lambda item: list(item) if isinstance(item, set) else item)
searxng_  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/json/__init__.py", line 238, in dumps
searxng_  |     **kw).encode(obj)
searxng_  |           ^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/json/encoder.py", line 200, in encode
searxng_  |     chunks = self.iterencode(o, _one_shot=True)
searxng_  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
searxng_  |   File "/usr/lib/python3.11/json/encoder.py", line 258, in iterencode
searxng_  |     return _iterencode(o, 0)
searxng_  |            ^^^^^^^^^^^^^^^^^
searxng_  | ValueError: Circular reference detected
searxng_  | [pid: 10|app: 0|req: 5/19] 127.0.0.1 () {24 vars in 341 bytes} [Sat Jun 17 14:48:28 2023] GET /search?q=test&format=json&categories=videos => generated 265 bytes in 1657 msecs (HTTP/1.1 500) 9 headers in 766 bytes (1 switches on core 1)

Any help will be appreciated :)
Thanks!
@mbaroncini mbaroncini added the bug Something isn't working label Jun 17, 2023
return42 added a commit to return42/searxng that referenced this issue Jun 19, 2023
@return42
[fix] implement a JSONEncoder for the json format
fab725b 
This patch implements a simple JSONEncoder just to fix searxng#2502 / on the long term
SearXNG needs a data schema for the result items and a json generator for the
result list.

Closes: searxng#2505
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@return42 return42 mentioned this issue Jun 19, 2023
Merged
@return42
Copy link
Member

@mbaroncini thanks a lot for your bug report and the detailed analysis that was very helpful for me 👍 .. I implemented a patch in

return42 added a commit that referenced this issue Jun 19, 2023
@return42
[fix] implement a JSONEncoder for the json format
86db087 
This patch implements a simple JSONEncoder just to fix #2502 / on the long term
SearXNG needs a data schema for the result items and a json generator for the
result list.

Closes: #2505
Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
@mbaroncini
Copy link
Author

Thanks @return42 !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[fix] implement a JSONEncoder for the json format return42/searxng
2 participants
@return42 @mbaroncini