ascii codec Error uploading CSV

[simonw]

Got this uploading CSV of the Squirrel Census:

{
	"0": {
		"id": "f029f3d9-1d9c-43f7-9eaa-297d3e3e3f9d",
		"filename": "2018_Central_Park_Squirrel_Census_-_Squirrel_Data",
		"bytes_todo": 747760,
		"bytes_done": 7604,
		"rows_done": 30,
		"started": "2022-07-03 20:10:18.452435",
		"completed": null,
		"error": "'ascii' codec can't decode byte 0xe2 in position 170: ordinal not in range(128)"
	}
}

[simonw]

2018_Central_Park_Squirrel_Census_-_Squirrel_Data.csv

[simonw]

This should have been fixed by the 0.7.1 release but apparently not!

• UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 105: invalid start byte #23

[simonw]

Hunch: this is because I only use the first 2MB to detect the encoding.

datasette-upload-csvs/datasette_upload_csvs/__init__.py

Lines 75 to 79 in 383014c

	# Use the first 2MB to detect the character encoding
	first_bytes = csv.file.read(2048)
	csv.file.seek(0)
	encoding = detect(first_bytes)["encoding"]
	print(encoding)

Also: is ascii a good default encoding to use? If the detection against the first 2MB works maybe I should always use latin-1 just in case?

[simonw]

Hah, spotted the other problem:

datasette-upload-csvs/datasette_upload_csvs/__init__.py

Lines 75 to 76 in 383014c

	# Use the first 2MB to detect the character encoding
	first_bytes = csv.file.read(2048)

The comment says 2MB but 2048 bytes is 2KB!