Skip to content
Permalink
Browse files

Fix issue streaming file upload into CSV parser.

Specifically, Flask (via werkzeug), uses a SpooledTemporaryFile to store
the upload data. This file is opened in mode "wb+", which isn't
compatible with the csv reader. TextIOWrapper can be used to translated
the data, but the fucking SpooledTemporaryFile doesn't implement the
fucking IOBase APIs. The issue is being bikeshedded by some assclown
whose avatar is literally an ass: python/cpython#3249
  • Loading branch information...
coleifer committed Mar 23, 2019
1 parent 1775491 commit 945d8d5ba29a520510db4007c148e42edb04638d
Showing with 30 additions and 9 deletions.
  1. +30 −9 sqlite_web/sqlite_web.py
@@ -13,20 +13,23 @@
from collections import namedtuple, OrderedDict
from functools import wraps
from getpass import getpass
from io import TextIOWrapper

# Py3k compat.
if sys.version_info[0] == 3:
binary_types = (bytes, bytearray)
decode_handler = 'backslashreplace'
numeric = (int, float)
unicode_type = str
from io import StringIO
else:
# Py2k compat.
if sys.version_info[0] == 2:
PY2 = True
binary_types = (buffer, bytes, bytearray)
decode_handler = 'replace'
numeric = (int, long, float)
unicode_type = unicode
from StringIO import StringIO
else:
PY2 = False
binary_types = (bytes, bytearray)
decode_handler = 'backslashreplace'
numeric = (int, float)
unicode_type = str
from io import StringIO

try:
from flask import (
@@ -555,12 +558,30 @@ def table_import(table):
format = 'json'
else:
format = 'csv'

# Here we need to translate the file stream. Werkzeug uses a
# spooled temporary file opened in wb+ mode, which is not
# compatible with Python's CSV module. We'd need to reach pretty
# far into Flask's internals to modify this behavior, so instead
# we'll just translate the stream into utf8-decoded unicode.
if not PY2:
try:
stream = TextIOWrapper(file_obj, encoding='utf8')
except AttributeError:
# The SpooledTemporaryFile used by werkzeug does not
# implement an API that the TextIOWrapper expects, so we'll
# just consume the whole damn thing and decode it.
# Fixed in werkzeug 0.15.
stream = StringIO(file_obj.read().decode('utf8'))
else:
stream = file_obj.stream

try:
with dataset.transaction():
count = dataset.thaw(
table,
format=format,
file_obj=file_obj.stream,
file_obj=stream,
strict=strict)
except Exception as exc:
flash('Error importing file: %s' % exc, 'danger')

2 comments on commit 945d8d5

@abathur

This comment has been minimized.

Copy link

replied Mar 23, 2019

@coleifer Not sure if it'll be a viable solution in your case, and I haven't attempted to use it yet, but Werkzeug's local workaround (pallets/werkzeug#1409) just got released in versions 0.15.* this week.

Saw a commit from another project excluding the bad versions, which seems prudent if this behavior is critical: kgaughan/docserver@8df45c1

Your commit message gave me a chuckle.

@coleifer

This comment has been minimized.

Copy link
Owner Author

replied Mar 23, 2019

The AttributeError is caught (it is raised if you're using the affected werkzeug versions), and a fallback is provided so I don't think I want to add such exclusions to the deps.

Please sign in to comment.
You can’t perform that action at this time.