UTF-8 encoded strings

[mre]

I'm writing a json-module compatible library for encoding and decoding JSON.
The following input string works with json, but not with my module:

    def test_decodeEscape(self):
        base = '\u00e5'.encode('utf-8')
        quote = "\"".encode()
        input = quote + base + quote
        print(input)
        json.loads(input)
>       mymodule.loads(input)
E       TypeError

💥 Reproducing

This repository contains a demonstration of the issue.
The loads function takes a &str as an input parameter. Maybe I have to change that to a different type in order to avoid the TypeError or maybe cast it to a different type.

🌍 Environment

• Operating system and version: MacOS X 10.12
• Python version: 3.6.5
• Installation: brew, no virtualenv
• Rust version: rustc 1.28.0-nightly (5bf68db6e 2018-05-28)

Any idea would be very much appreciated.

[althonos]

It is not that your function does not accept UTF-8 encoded strings, it is that it does accept bytestrings (ìt will fail with mymodule.loads(b'abc)').

If you want to accept only bytestrings, use PyBytes as your function argument; if you want only strings, use &str or PyString; if you want to enforce unicode strings (u'abc') in Python 2, use PyUnicode; if you want to support several types of arguments, you need to accept PyObject as your function argument and typecheck dynamically yourself (I guess ?).

You could also probably do the type check on the Python side, and use functools.singledispatch to map to the correct underlying function (like _load_str and _load_bytes for instance).

[althonos]

Kudos for the test repository, that's how I figured it out ! 😉

[mre]

Thanks! I'll try that. Just saw that the official json module also accepts a PyObject it seems.

[althonos]

If I'm not mistaken, all CPython functions do because there's no downcast from Python int to C int for instance, so all functions do and downcasting is done in the function body (one of the things that's handled for you in most cases in pyo3). If you try json.loads(1) you'll still get a TypeError because only str, bytes and bytearray objects are accepted by that function!