Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[flickr_noapi] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) #1799

Open
Perflyst opened this issue Jan 7, 2020 · 9 comments

Comments

@Perflyst
Copy link

@Perflyst Perflyst commented Jan 7, 2020

Similar to #419

Installation: current master commit
How to reproduce? Search for "kek" on https://search.snopyta.org/ and click on "Images"

ERROR:flask.app:Exception on / [POST]
Traceback (most recent call last):
  File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/searx/searx-ve/local/lib/python2.7/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/searx/searx/webapp.py", line 544, in index
    result['title'] = highlight_content(escape(result['title'] or u''), search_query.query)
  File "/usr/local/searx/searx/utils.py", line 79, in highlight_content
    if content.lower().find(query.lower()) > -1:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
@unixfox

This comment has been minimized.

Copy link
Contributor

@unixfox unixfox commented Jan 7, 2020

Probably related to the engine "flickr" because that's the only one doing this error on search.snopyta.org but I can't reproduce the same bug with "flickr" engine enabled by default on my instance: https://searx.be/?q=kek&time_range=&language=en-US&category_images=on

Maybe flickr gives back results with non unicode due to the location of the instance?

@Yetangitu

This comment has been minimized.

Copy link
Contributor

@Yetangitu Yetangitu commented Jan 7, 2020

This mostly likely is a double-encoding error in flickr_noapi.py which can be fixed by removing the calls to encode('utf-8'), like so:

diff --git a/searx/engines/flickr_noapi.py b/searx/engines/flickr_noapi.py
index e1abb378..c8ee34f7 100644
--- a/searx/engines/flickr_noapi.py
+++ b/searx/engines/flickr_noapi.py
@@ -118,9 +118,9 @@ def response(resp):
             'template': 'images.html'
         }
         try:
-            result['author'] = author.encode('utf-8')
-            result['title'] = title.encode('utf-8')
-            result['content'] = content.encode('utf-8')
+            result['author'] = author
+            result['title'] = title
+            result['content'] = content
         except:
             result['author'] = ''
             result['title'] = ''
@asciimoo asciimoo closed this in db9d7d4 Jan 8, 2020
asciimoo added a commit that referenced this issue Jan 8, 2020
[engines|fix] flickr_noapi: Fix double-encode error (fixes #1799)
@bauruine

This comment has been minimized.

Copy link

@bauruine bauruine commented Jan 8, 2020

I added #1800 to my instance and it fixes kek but now I'm getting a different error for "!fl cat up" which works without the patch.

ERROR:flask.app:Exception on / [POST]
Traceback (most recent call last):
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/flask/app.py", line 1816, in full_dispatch_request
    return self.finalize_request(rv)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/flask/app.py", line 1831, in finalize_request
    response = self.make_response(rv)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/flask/app.py", line 1968, in make_response
    rv = self.response_class(rv, status=status, headers=headers)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/werkzeug/wrappers.py", line 862, in __init__
    self.set_data(response)
  File "/usr/local/searx/searx-ve/lib/python3.6/site-packages/werkzeug/wrappers.py", line 1002, in set_data
    value = value.encode(self.charset)
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 27080-27081: surrogates not allowed

@Perflyst

This comment has been minimized.

Copy link
Author

@Perflyst Perflyst commented Jan 8, 2020

that does not happen for me @bauruine, but the "kek" bug is also fixed for me

@Perflyst

This comment has been minimized.

Copy link
Author

@Perflyst Perflyst commented Jan 8, 2020

Just updated from python2.7 to python3.5, now I get the same 500 error as @bauruine

@return42

This comment has been minimized.

Copy link
Collaborator

@return42 return42 commented Jan 13, 2020

Sorry for the late reply .. are you still have such errors? Is it related to python3 or a search engine?

I guess we have to fix more engines that way .. if you see any similar, please add a new issue / thanks!

@return42 return42 changed the title UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) [flickr_noapi] UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) Jan 13, 2020
@asciimoo

This comment has been minimized.

Copy link
Owner

@asciimoo asciimoo commented Jan 13, 2020

#1800 only fixes engines/flickr_noapi.py

The problem is still there in flickr and needs a more sophisticated solution. Flickr sends meta data in different encodings.

Python 2 is no longer supported

I think we should still support py2 at least while major distros support it.

@return42 return42 reopened this Jan 13, 2020
@return42

This comment has been minimized.

Copy link
Collaborator

@return42 return42 commented Jan 13, 2020

I think we should still support py2 at least while major distros support it.

They also ship py3 .. There will always be some distros that do not switch to py3 by default.

Python 2 is started 20 years ago, Python 3 started in Dec. 2008, Python 2.7's end-of-life date was initially set at 2015 then postponed to 2020 .. in 2020 I really want to get rid of the py2/3 mess, I'm so sick of py2 support in all the projects I support.

Py3 should the default installation method, if someone does not have py3 he should use containers .. there are so many solution.

@asciimoo

This comment has been minimized.

Copy link
Owner

@asciimoo asciimoo commented Jan 13, 2020

I'll create a poll, lets see what is the opinion of the admins/users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.