Encoding

[roramirez]

When I used make_response_from_query_sets from a database, if some columns have encoding like iso-8859-1 is broken. There any way for set the enconding type?

The log is

Traceback (most recent call last):
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/enquesta/app.py", line 57, in operation_export
    file_name="operation_data")
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_webio/__init__.py", line 322, in make_response_from_query_sets
    dest_file_type=file_type, **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/core.py", line 161, in save_as
    return sources.save_sheet(sheet, **dest_keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/__init__.py", line 33, in save_sheet
    source.write_data(sheet)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/file_source_output.py", line 77, in write_data
    sheet, **self.keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/renderers/_excel.py", line 36, in render_sheet_to_stream
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 70, in save_data
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 92, in store_data
    writer.close()
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_xls/xls.py", line 243, in close
    self.work_book.save(self.file_alike_object)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 696, in save
    doc.save(filename_or_stream, self.get_biff_data())
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 660, in get_biff_data
    shared_str_table   = self.__sst_rec()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 622, in __sst_rec
    return self.__sst.get_biff_record()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 77, in get_biff_record
    self._add_to_sst(s)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 92, in _add_to_sst
    u_str = upack2(s, self.encoding)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/UnicodeUtils.py", line 50, in upack2
    us = unicode(s, encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 8: ordinal not in range(128)

[chfw]

A custom row renderer is provided to help with this issue. You could use it like the following:

    elif atype == "custom":
        question = Question.objects.get(slug='ide')
        query_sets = Choice.objects.filter(question=question)
        column_names = ['choice_text', 'id', 'votes']
        def row_renderer(row):
            # you probably want to convert the latin column to utf8
            return [str(e)+'e' for e in row] # an example only
        return excel.make_response_from_query_sets(
            query_sets,
            column_names,
            'xls',
            file_name="custom",
            row_renderer=row_renderer
        )

In order to evaluate the feature, you need to do these first:

pip install https://github.com/pyexcel/pyexcel-io/archive/master.zip # unreleased 0.2.3
pip install https://github.com/pyexcel/pyexcel/archive/master.zip # with previous commit

[roramirez]

Yeah!, is working that way. Thanks.

Do you think there any way for overwrite the encoding option of xlwt?

[chfw]

pyexcel-xls use 'ascii' as default encoding:

    def open(self, file_name,
             encoding='ascii', style_compression=2, **keywords):
        BookWriter.open(self, file_name, **keywords)
        self.work_book = Workbook(style_compression=style_compression,
                                  encoding=encoding)

However, you can override it using dest_encoding="utf-8". As in previous example, you could append one extra line:

...
        return excel.make_response_from_query_sets(
            query_sets,
            column_names,
            'xls',
            file_name="custom",
            row_renderer=row_renderer,
            dest_encoding="utf-8" # for example
        )

[roramirez]

@chfw Thanks a lot!