api cannot handle unicode

[ragesoss]

For example: http://tools.wmflabs.org/eranbot/plagiabot/api.py?action=suspected_diffs&page_title=%27Ch%C3%A2teau%20Laroque%27

[ragesoss]

As far as I can tell, there's no way to query for articles with titles like this. Here's the entry returned by the database for the general 50-limit query:

{'page_ns': '0', 'page_title': 'Ch\xc3\xa2teau_Laroque', 'diff_timestamp': '20150831161600', 'ithenticate_id': '19202899', 'project': 'wikipedia', 'diff': '678784050'}

But passing that same page_title page yields no results: http://tools.wmflabs.org/eranbot/plagiabot/api.py?action=suspected_diffs&lang=en&page_title=Ch\xc3\xa2teau_Laroque

[valhallasw]

What happens is the following:

• %27Ch%C3%A2teau%20Laroque%27 is passed through the url
• parse_qs parses this in "'x27Ch\xc3\xa2teau Laroque'"
• this is a non-ascii byte string, and still needs to be decoded to utf-8 before passing it to oursql.

[ragesoss]

I'm not sure how tied to pywikibot this still is, but if the api portion (if not all of it) could be switched to python3, that'd probably involve a lot less unicode pain in the long run.

[fhocutt]

This is causing issues with the integration into @earwig's Copyvios Detector: earwig/copyvios#25
The API should at least handle errors and output a documented warning message instead of serving the html stack trace.

[eranroz]

Solved using ugly decode.

Using python3 is a good idea, but the instance in labs uses python2 and I'm too lazy to move the instance to use python3.

[eranroz]

tested:
http://tools.wmflabs.org/eranbot/plagiabot/api.py?action=suspected_diffs&lang=en&page_title=Ch%C3%A2teau_Laroque

[earwig]

Cheers!

[kaldari]

Is this ticket resolved now?

[eranroz]

yes