Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Encoding #59

Closed
roramirez opened this issue Sep 14, 2016 · 4 comments
Closed

Encoding #59

roramirez opened this issue Sep 14, 2016 · 4 comments

Comments

@roramirez
Copy link

@roramirez roramirez commented Sep 14, 2016

When I used make_response_from_query_sets from a database, if some columns have encoding like iso-8859-1 is broken. There any way for set the enconding type?

The log is


Traceback (most recent call last):
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/root/enquesta/env/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/root/enquesta/app.py", line 57, in operation_export
    file_name="operation_data")
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_webio/__init__.py", line 322, in make_response_from_query_sets
    dest_file_type=file_type, **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/core.py", line 161, in save_as
    return sources.save_sheet(sheet, **dest_keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/__init__.py", line 33, in save_sheet
    source.write_data(sheet)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/sources/file_source_output.py", line 77, in write_data
    sheet, **self.keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel/renderers/_excel.py", line 36, in render_sheet_to_stream
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 70, in save_data
    **keywords)
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_io/io.py", line 92, in store_data
    writer.close()
  File "/root/enquesta/env/lib/python2.7/site-packages/pyexcel_xls/xls.py", line 243, in close
    self.work_book.save(self.file_alike_object)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 696, in save
    doc.save(filename_or_stream, self.get_biff_data())
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 660, in get_biff_data
    shared_str_table   = self.__sst_rec()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/Workbook.py", line 622, in __sst_rec
    return self.__sst.get_biff_record()
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 77, in get_biff_record
    self._add_to_sst(s)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/BIFFRecords.py", line 92, in _add_to_sst
    u_str = upack2(s, self.encoding)
  File "/root/enquesta/env/lib/python2.7/site-packages/xlwt/UnicodeUtils.py", line 50, in upack2
    us = unicode(s, encoding)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 8: ordinal not in range(128)
@chfw
Copy link
Member

@chfw chfw commented Sep 15, 2016

A custom row renderer is provided to help with this issue. You could use it like the following:

    elif atype == "custom":
        question = Question.objects.get(slug='ide')
        query_sets = Choice.objects.filter(question=question)
        column_names = ['choice_text', 'id', 'votes']
        def row_renderer(row):
            # you probably want to convert the latin column to utf8
            return [str(e)+'e' for e in row] # an example only
        return excel.make_response_from_query_sets(
            query_sets,
            column_names,
            'xls',
            file_name="custom",
            row_renderer=row_renderer
        )

In order to evaluate the feature, you need to do these first:

pip install https://github.com/pyexcel/pyexcel-io/archive/master.zip # unreleased 0.2.3
pip install https://github.com/pyexcel/pyexcel/archive/master.zip # with previous commit
@roramirez
Copy link
Author

@roramirez roramirez commented Sep 16, 2016

Yeah!, is working that way. Thanks.

Do you think there any way for overwrite the encoding option of xlwt?

@chfw
Copy link
Member

@chfw chfw commented Sep 17, 2016

pyexcel-xls use 'ascii' as default encoding:

    def open(self, file_name,
             encoding='ascii', style_compression=2, **keywords):
        BookWriter.open(self, file_name, **keywords)
        self.work_book = Workbook(style_compression=style_compression,
                                  encoding=encoding)

However, you can override it using dest_encoding="utf-8". As in previous example, you could append one extra line:

...
        return excel.make_response_from_query_sets(
            query_sets,
            column_names,
            'xls',
            file_name="custom",
            row_renderer=row_renderer,
            dest_encoding="utf-8" # for example
        )
@roramirez
Copy link
Author

@roramirez roramirez commented Sep 17, 2016

@chfw Thanks a lot!

@roramirez roramirez closed this Sep 17, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.