New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utility functions fieldnames() and dicts() don't support unicode fieldnames #350

sv1jsb opened this Issue Dec 2, 2015 · 4 comments


None yet
2 participants

sv1jsb commented Dec 2, 2015

First of all thank you very much for petl. Very useful.

I am using petl to manipulate csv files from excel.
Loading is done with the fromcsv() function.
Columns have titles with non ASCII characters and file encoding is utf-8.
Loading is done flawlessly, but when I try to get the table as a list of dicts,
with the help petl.dicts() function, I get the well known error:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-6: ordinal not in range(128)

I have narrowed down the problem to the base.asdicts and base.fieldnames functions which
use the str() function and not the unicode(), while iterating through the header fields.

I am using python 2.7.

Best regards,


This comment has been minimized.


alimanfoo commented Dec 2, 2015

Thanks Andreas, this sounds like a simple fix. I'll put this high priority for the next release, will hopefully get to that soon.

@alimanfoo alimanfoo added this to the v1.1 milestone Dec 2, 2015


This comment has been minimized.


sv1jsb commented Dec 2, 2015

Thanks Alistair,
I can help , if you 'd like, making a pull request.
I will only change the aforementioned lines.


This comment has been minimized.


alimanfoo commented Dec 2, 2015

Thanks Andreas, a PR would be very welcome. I think it would be best to use
text_type() imported from petl.compat rather than unicode() as this should
work on both PY2 and PY3.

On Wednesday, December 2, 2015, Andreas Porevopoulos <> wrote:

Thanks Alistair,
I can help , if you 'd like, making a pull request.
I will only change the aforementioned lines.

Reply to this email directly or view it on GitHub
#350 (comment).

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
United Kingdom
Tel: +44 (0)1865 287721


This comment has been minimized.


sv1jsb commented Dec 2, 2015

Of course.

sv1jsb added a commit to sv1jsb/petl that referenced this issue Dec 2, 2015

@alimanfoo alimanfoo closed this in 74990c4 Dec 2, 2015

alimanfoo added a commit that referenced this issue Dec 2, 2015

andrewakim added a commit to andrewakim/petl that referenced this issue Jul 13, 2016

Pulling in changes and updating my fork of petl (#1)
* work on PY3 compatibility, working through python 3 test failures, some way to go yet

* remove multirangeaggregate; work on flexible comparisons

* fix errors in joins

* work on csv compatibility, unicode todo

* working on csv compatibility, part-baked

* closer to working csv module

* csv module complete?

* csv all working

* rewrite text module

* various fixes

* all tests passing in py26, py27, py34

* dbtests working

* doco

* added support for write_header in tee(u)csv

* docs reworked; consistent state

* review documentation

* doctests passing

* doctests; change select behaviour for relaxed comparison

* doctests pass

* doctests pass

* petl.transform.basics doctests pass

* petl.transform.conversions doctests pass

* petl.transform.dedup doctests pass

* petl.transform.conversions doctests pass

* reorganise project layout

* petl.transform.maps doctests passing

* petl.transform.reductions doctests passing

* petl.transform.regex doctests passing

* petl.transform.reshape doctests passing

* petl.transform.selects doctests passing

* petl.transform.setops doctests passing

* petl.transform.sorts and unpacks doctests passing

* refactor petl.util

* fix tests

* combine fluent and interactive into main petl

* complete integration of fluent and interactive into core

* doco rewrite; doctest rewrite

* all doctests passing

* all tests passing

* integrate petlx.display; rework case study 1; add html support for styling; review docs

* doco review; document v1.0 changes

* remove u... functions

* fix docs after removing u... functions

* rework config and vis

* rerun reprhtml nb

* doctests working again after look changes

* v1.0.0-alpha1

* review documentation; change some parameter names

* doco

* v1.0.0a2

* increment version

* doco

* consistify variable naming

* consistify variable naming

* consistify variable names

* doco

* doco

* refactor dbtests

* fix petl executable

* add contributing doc; fix case study

* doco

* v1.0.0a3

* increment version

* resolves petl-developers#303

* resolves petl-developers#295

* resolves petl-developers#294

* various fixes

* closes petl-developers#205

* closes petl-developers#142; closes petl-developers#293 by merging in changes manually

* port array module from petlx

* port array module complete

* migrate dataframe, xls, xlsx

* port hdf5

* doco

* tox running

* merge sqlite3 into db module; migrate db_create from petlx

* doco

* migrate intervals

* doco

* rename modules with optional dependencies; migrate whoosh module

* documentation

* add validation module; add truncate and width to look(); add truncate to display(); rename some variables

* fix doctests

* v1.0.0a4

* increment version

* doco, prep for beta

* v1.0.0b1

* bump version

* add config.look_width

* windows setup

* test_csv pass windows

* test_csv_unicode pass windows

* test_html pass windows

* test_html_unicode pass windows

* fix html tests for cross-platform line endings

* fix test_sources for windows

* fix test_sqlite3 for windows

* fix test_tees for windows

* fix test_text for windows

* fix test_text_unicode for windows

* add excel test resources

* fix missing resources

* test compatibility fixes

* fix excel tests for windows

* fix sort file-caching to work on windows

* test sort cleanup

* fix tempdir in sort tests

* add logging

* get sort cleanup working on windows

* fix test compatibility for windows

* closes petl-developers#245 by moving import sqlite3 inside function, and trying import of cStringIO and cPickle falling back to import StringIO and pickle under PY2

* doco; add finer-grained exception

* v1.0.0b2

* v1.0.0

* minor doco

* doco

* rerun case study

* add tests for petl-developers#308, experiment with implementation

* mid-work

* fixes petl-developers#308

* doco

* v1.0.1

* logging

* add pypy support

* closes petl-developers#309

* v1.0.2

* Catches ValueError thrown when sys.stdin is a pseudofile

When using py.test it redirect stdin/stdout in pseudofiles by
default. Pseudofile do not provide a fileno() method and as such
raise a `ValueError` as soon as `` is imported (as shown

tests/transform/ in <module>
    import petl
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/ in <module>
    from petl import util
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/ in <module>
    from petl.util.vis import look, lookall, lookstr, lookallstr, see
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/ in <module>
    from import MemorySource
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in <module>
    from import FileSource, GzipSource, BZ2Source, ZipSource, \
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in <module>
    stdin_binary = _get_stdin_binary()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in _get_stdin_binary
    fd = sys.stdin.fileno()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/_pytest/ in fileno
    raise ValueError("redirected Stdin is pseudofile, has no fileno()")
E   ValueError: redirected Stdin is pseudofile, has no fileno()

Simply adding `ValueError` to the tuple of caught exceptions resolves the issues.

* v1.0.3

* doco

* v1.0.4

* closes petl-developers#318

* v1.0.5

* doco

* Use lxml if available to enable parent traversal in xml source

* closes petl-developers#315

* micro version bump

* PEP8; doco

* ack

* doco

* doco

* doco

* doco

* resolve conflict example

* example

* fixes petl-developers#324

* v1.0.7

* Adds support for named paramstyle

* Fixes petl.tocsv() exporting of 'None' string
Petl tocsv exports 'None' instad of no value if used with python 2.x

* add tox

* fix xlsx tests for recent version of openpyxl

* pin versions for optional requirements

* pin sphinx version for rtfd

* petl-developers#326 add documentation note on how to work around cx_Oracle limitation

* micro version bump

* add contributors

* closes petl-developers#325 with tests and fix

* v1.0.9

* closes petl-developers#317 by removing IF EXISTS and trailing semicolon

* v1.0.10

* fix petl-developers#330

* test for petl-developers#330. close petl-developers#330

* Adds test for commit 660bc12
Test `test_tocsv_none()` is added that validates whether `tocsv()` with `None` elements generates empty CSV values.

* Added unit test to cover renaming multiple headers by index

* closes petl-developers#335 with minor changes after PR

* ack

* v1.0.11

* add "strict" keyword doc for rename() function

* ack psnj

* fix for new tox

* listify incoming iterable of dicts to fix petl-developers#341

It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys-for-the-header implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header.

* Per email from Brad Maggard on 2015-09-16, addfieldusingcontext() doesn't have access to the new field, making calculations like running sum more complicated.

The generator yields the tuple from the original table.  It now creates a new Record() object with the additional field and the previous row's value, providing access to the prior row's new field.

* upgrade requirements

* fix doctests for numpy upgrade

* fix for petl-developers#350

* following on from petl-developers#350, ensure all handling of field names is via text_type

* some work on petl-developers#349

* fix petl-developers#347 in general by changing default value for errors kwarg

* close petl-developers#349

* newline

* work on petl-developers#345

* work on petl-developers#345

* work on petl-developers#345

* fixes petl-developers#345

* extension to original fix for petl-developers#341 to avoid complete listification

* fixes petl-developers#209

* fixes petl-developers#332; also adds sortheader() function

* fixes petl-developers#339 by adding biselect

* add sortheader() as Table method

* fix tox for platform differences

* resolves petl-developers#337 by adding epilogue argument

* remove test requirements for pypy

* resolves petl-developers#327 via addition of stack() method; also revisit cat()

* doco; missing method

* resolves petl-developers#323 with documentation and extra test

* doco

* resolves petl-developers#316 with addition of fromcolumns()

* initial work on to/from bcolz petl-developers#310

* resolves petl-developers#310

* added change note

* resolves petl-developers#333 with strict parameter to complement() etc.

* resolves petl-developers#319 with new function groupselectlast

* resolves petl-developers#256 with example in docstring

* doco

* doco

* doco

* Utilize zip_longest to mirror behavior in fromdicts(), filling in None when lengths differ

* make missing value configurable

* fix tests for change in field ordering behaviour

* fix doctest

* fix release notes

* v1.1.0

* fix contributor link

* add field name argument to addrownumbers() to close petl-developers#366

* corrected comments for groupselectmin and groupselectmax

* Fix link in documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment