UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 105: invalid start byte

[simonw]

Got this uploading this CSV: https://static.simonwillison.net/static/2022/Animal%20Rescue%20incidents%20attended%20by%20LFB%20from%20Jan%202009.csv

Relevant TODO:

datasette-upload-csvs/datasette_upload_csvs/__init__.py

Lines 92 to 96 in 8127afc

	def insert_docs(conn):
	database = sqlite_utils.Database(conn)
	# TODO: Support other encodings:
	reader = csv_std.reader(codecs.iterdecode(csv.file, "utf-8"))
	headers = next(reader)

[simonw]

Related issue: if this happens an error message is shown in the logs but not to the user.

[simonw]

I don't want to tell users to select the encoding themselves before upload, which means I need to detect it.

It looks like this is the modern alternative to chardet: https://charset-normalizer.readthedocs.io/en/latest/