New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

can not look the return result with fromcsv? #347

Closed
Alwayslow opened this Issue Oct 22, 2015 · 6 comments

Comments

Projects
None yet
3 participants
@Alwayslow

Alwayslow commented Oct 22, 2015

windows 7 32bit python 2.7.10

import csv
with open('test.csv', 'wb') as f:
writer = csv.writer(f)
writer.writerow(['foo', 'bar'])
writer.writerow(['a', 1])
writer.writerow(['b', 2])
writer.writerow(['c', 2])

from petl import fromcsv, look
testcsv = fromcsv('test.csv')
look(testcsv)

Traceback (most recent call last):
File "<pyshell#8>", line 1, in
look(testcsv)
File "D:\ProgramFiles\python27\lib\site-packages\petl\util\vis.py", line 104, in repr
table, overflow = _vis_overflow(self.table, self.limit)
File "D:\ProgramFiles\python27\lib\site-packages\petl\util\vis.py", line 528, in _vis_overflow
table = list(islice(table, 0, limit+2))
File "D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line 44, in iter
errors=self.errors, **self.csvargs)
File "D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line 145, in init
f = UTF8Recoder(f, encoding=encoding, errors=errors)
File "D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line 133, in init
self.reader = codec.streamreader(buf, errors=errors)
TypeError: StreamReader() argument 2 must be string, not None

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Oct 22, 2015

Thanks for the bug report, I'll look into it.

On Thu, Oct 22, 2015 at 6:07 AM, Alwayslow notifications@github.com wrote:

windows 7 32bit python 2.7.10

import csv
with open('test.csv', 'wb') as f:
writer = csv.writer(f)
writer.writerow(['foo', 'bar'])
writer.writerow(['a', 1])
writer.writerow(['b', 2])
writer.writerow(['c', 2])

from petl import fromcsv, look
testcsv = fromcsv('test.csv')
look(testcsv)

Traceback (most recent call last):
File "", line 1, in
look(testcsv)
File "D:\ProgramFiles\python27\lib\site-packages\petl\util\vis.py", line
104, in repr
table, overflow =

_vis_overflow(self.table, self.limit) File
"D:\ProgramFiles\python27\lib\site-packages\petl\util\vis.py", line 528, in
vis_overflow table = list(islice(table, 0, limit+2)) File
"D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line 44,
in _iter
errors=self.errors, *self.csvargs)
File "D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line
145, in *init

f = UTF8Recoder(f, encoding=encoding, errors=errors)
File "D:\ProgramFiles\python27\lib\site-packages\petl\io\csv_py2.py", line
133, in init
self.reader = codec.streamreader(buf, errors=errors)
TypeError: StreamReader() argument 2 must be string, not None


Reply to this email directly or view it on GitHub
#347.

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://purl.org/net/aliman
Email: alimanfoo@googlemail.com alimanfoo@gmail.com
Tel: +44 (0)1865 287721

@Alwayslow

This comment has been minimized.

Alwayslow commented Oct 23, 2015

I have solve the problem,on windows ,you must append the encoding argument in fromcsv()、tocsv() and so on:
tocsv(rowiteror_list, 'example.csv', encoding='utf-8')
tab1_1 = fromcsv('example.csv', encoding='utf-8')
print look(tab1_1)

@alimanfoo alimanfoo closed this Nov 25, 2015

@loadingxys

This comment has been minimized.

loadingxys commented Dec 3, 2015

I have met the same question when i was using .display() in "petl Case Study 1 - Comparing Tables"

import petl as etl
a = etl.fromtsv('snpdata.csv',encoding='utf-8')
b = etl.fromtsv('popdata.csv',encoding='utf-8')
b_renamed = b.rename({'Chromosome': 'Chr', 
                      'Coordinates': 'Pos', 
                      'Ref. Allele': 'Ref', 
                      'Non-Ref. Allele': 'Nref', 
                      'Derived Allele': 'Der', 
                      'Mutation type': 'Mut', 
                      'Gene': 'GeneId', 
                      'Gene Aliases': 'GeneAlias', 
                      'Gene Description': 'GeneDescr'})
common_fields = ['Chr', 'Pos', 'Ref', 'Nref', 'Der', 'Mut', 'GeneId', 'GeneAlias', 'GeneDescr']
a_common = a.cut(common_fields)
b_common = b_renamed.cut(common_fields)

"""here is a problem !!!"""
print a_common.display(vrepr=repr)

and then the error is:

Traceback (most recent call last):    print a_common.display(vrepr=repr)
  File "D:\Tools\python\lib\site-packages\petl\util\vis.py", line 575, in display
    encoding=encoding, truncate=truncate)
  File "D:\Tools\python\lib\site-packages\petl\util\vis.py", line 552, in _display_html
    td_styles=td_styles, truncate=truncate)
  File "D:\Tools\python\lib\site-packages\petl\io\html.py", line 66, in tohtml
    f = codec.streamwriter(buf, errors=errors)
TypeError: StreamWriter() argument 2 must be string, not None

i tried to add encoding in .display() like this:

print a_common.display(vrepr=repr,encoding='utf-8')

and then, it returns 'None'

None
[Finished in 1.3s]

where is wrong ?
thanks

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Dec 3, 2015

Hi @loadingxys, the display() functions is meant for use within a Jupyter (IPython) notebook, have you tried it there? It generates an HTML representation of a table and then displays it within the notebook. There are some examples here: http://nbviewer.ipython.org/github/alimanfoo/petl/blob/master/repr_html.ipynb

More generally it is annoying that on Windows errors like this arise, I will see if anything can be done to avoid passing in a None encoding.

@alimanfoo alimanfoo added this to the v1.1 milestone Dec 3, 2015

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Dec 3, 2015

Just to correct, looks like problem is with None value being passed as the 'errors' argument, nothing to do with encodings.

alimanfoo added a commit that referenced this issue Dec 3, 2015

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Dec 3, 2015

I have changed the default value for the errors kwarg to 'strict' instead of None, which should hopefully fix issues like this in general. Closing as should prevent the reported errors arising but please re-open if you cannot verify.

andrewakim added a commit to andrewakim/petl that referenced this issue Jul 13, 2016

Pulling in changes and updating my fork of petl (#1)
* work on PY3 compatibility, working through python 3 test failures, some way to go yet

* remove multirangeaggregate; work on flexible comparisons

* fix errors in joins

* work on csv compatibility, unicode todo

* working on csv compatibility, part-baked

* closer to working csv module

* csv module complete?

* csv all working

* rewrite text module

* various fixes

* all tests passing in py26, py27, py34

* dbtests working

* doco

* added support for write_header in tee(u)csv

* docs reworked; consistent state

* review petl.io.pickle documentation

* petl.io.sqlite3 doctests passing

* petl.io.text doctests; change select behaviour for relaxed comparison

* petl.io.text doctests pass

* petl.io.xml doctests pass

* petl.transform.basics doctests pass

* petl.transform.conversions doctests pass

* petl.transform.dedup doctests pass

* petl.transform.conversions doctests pass

* reorganise project layout

* petl.transform.maps doctests passing

* petl.transform.reductions doctests passing

* petl.transform.regex doctests passing

* petl.transform.reshape doctests passing

* petl.transform.selects doctests passing

* petl.transform.setops doctests passing

* petl.transform.sorts and unpacks doctests passing

* refactor petl.util

* fix tests

* combine fluent and interactive into main petl

* complete integration of fluent and interactive into core

* doco rewrite; doctest rewrite

* all doctests passing

* all tests passing

* integrate petlx.display; rework case study 1; add html support for styling; review docs

* doco review; document v1.0 changes

* remove u... functions

* fix docs after removing u... functions

* rework config and vis

* rerun reprhtml nb

* doctests working again after look changes

* v1.0.0-alpha1

* review documentation; change some parameter names

* doco

* v1.0.0a2

* increment version

* doco

* consistify variable naming

* consistify variable naming

* consistify variable names

* doco

* doco

* refactor dbtests

* fix petl executable

* add contributing doc; fix case study

* doco

* v1.0.0a3

* increment version

* resolves petl-developers#303

* resolves petl-developers#295

* resolves petl-developers#294

* various fixes

* closes petl-developers#205

* closes petl-developers#142; closes petl-developers#293 by merging in changes manually

* port array module from petlx

* port array module complete

* migrate dataframe, xls, xlsx

* port hdf5

* doco

* tox running

* merge sqlite3 into db module; migrate db_create from petlx

* doco

* migrate intervals

* doco

* rename modules with optional dependencies; migrate whoosh module

* documentation

* add validation module; add truncate and width to look(); add truncate to display(); rename some variables

* fix doctests

* v1.0.0a4

* increment version

* doco, prep for beta

* v1.0.0b1

* bump version

* add config.look_width

* windows setup

* test_csv pass windows

* test_csv_unicode pass windows

* test_html pass windows

* test_html_unicode pass windows

* fix html tests for cross-platform line endings

* fix test_sources for windows

* fix test_sqlite3 for windows

* fix test_tees for windows

* fix test_text for windows

* fix test_text_unicode for windows

* add excel test resources

* fix missing resources

* test compatibility fixes

* fix excel tests for windows

* fix sort file-caching to work on windows

* test sort cleanup

* fix tempdir in sort tests

* add logging

* get sort cleanup working on windows

* fix test compatibility for windows

* closes petl-developers#245 by moving import sqlite3 inside function, and trying import of cStringIO and cPickle falling back to import StringIO and pickle under PY2

* doco; add finer-grained exception

* v1.0.0b2

* v1.0.0

* minor doco

* doco

* rerun case study

* add tests for petl-developers#308, experiment with implementation

* mid-work

* fixes petl-developers#308

* doco

* v1.0.1

* logging

* add pypy support

* closes petl-developers#309

* v1.0.2

* Catches ValueError thrown when sys.stdin is a pseudofile

When using py.test it redirect stdin/stdout in pseudofiles by
default. Pseudofile do not provide a fileno() method and as such
raise a `ValueError` as soon as `petl.io.sources` is imported (as shown
below).

```
tests/transform/test_transforms.py:1: in <module>
    import petl
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/__init__.py:9: in <module>
    from petl import util
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/__init__.py:13: in <module>
    from petl.util.vis import look, lookall, lookstr, lookallstr, see
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/vis.py:12: in <module>
    from petl.io.sources import MemorySource
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/__init__.py:3: in <module>
    from petl.io.sources import FileSource, GzipSource, BZ2Source, ZipSource, \
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/sources.py:150: in <module>
    stdin_binary = _get_stdin_binary()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/sources.py:133: in _get_stdin_binary
    fd = sys.stdin.fileno()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/_pytest/capture.py:441: in fileno
    raise ValueError("redirected Stdin is pseudofile, has no fileno()")
E   ValueError: redirected Stdin is pseudofile, has no fileno()
```

Simply adding `ValueError` to the tuple of caught exceptions resolves the issues.

* v1.0.3

* doco

* v1.0.4

* closes petl-developers#318

* v1.0.5

* doco

* Use lxml if available to enable parent traversal in xml source

* closes petl-developers#315

* micro version bump

* PEP8; doco

* ack

* doco

* doco

* doco

* doco

* resolve conflict example

* example

* fixes petl-developers#324

* v1.0.7

* Adds support for named paramstyle

* Fixes petl.tocsv() exporting of 'None' string
Petl tocsv exports 'None' instad of no value if used with python 2.x

* add tox

* fix xlsx tests for recent version of openpyxl

* pin versions for optional requirements

* pin sphinx version for rtfd

* petl-developers#326 add documentation note on how to work around cx_Oracle limitation

* micro version bump

* add contributors

* closes petl-developers#325 with tests and fix

* v1.0.9

* closes petl-developers#317 by removing IF EXISTS and trailing semicolon

* v1.0.10

* fix petl-developers#330

* test for petl-developers#330. close petl-developers#330

* Adds test for commit 660bc12
Test `test_tocsv_none()` is added that validates whether `tocsv()` with `None` elements generates empty CSV values.

* Added unit test to cover renaming multiple headers by index

* closes petl-developers#335 with minor changes after PR

* ack

* v1.0.11

* add "strict" keyword doc for rename() function

* ack psnj

* fix for new tox

* listify incoming iterable of dicts to fix petl-developers#341

It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys-for-the-header implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header.

* Per email from Brad Maggard on 2015-09-16, addfieldusingcontext() doesn't have access to the new field, making calculations like running sum more complicated.

The generator yields the tuple from the original table.  It now creates a new Record() object with the additional field and the previous row's value, providing access to the prior row's new field.

* upgrade requirements

* fix doctests for numpy upgrade

* fix for petl-developers#350

* following on from petl-developers#350, ensure all handling of field names is via text_type

* some work on petl-developers#349

* fix petl-developers#347 in general by changing default value for errors kwarg

* close petl-developers#349

* newline

* work on petl-developers#345

* work on petl-developers#345

* work on petl-developers#345

* fixes petl-developers#345

* extension to original fix for petl-developers#341 to avoid complete listification

* fixes petl-developers#209

* fixes petl-developers#332; also adds sortheader() function

* fixes petl-developers#339 by adding biselect

* add sortheader() as Table method

* fix tox for platform differences

* resolves petl-developers#337 by adding epilogue argument

* remove test requirements for pypy

* resolves petl-developers#327 via addition of stack() method; also revisit cat()

* doco; missing method

* resolves petl-developers#323 with documentation and extra test

* doco

* resolves petl-developers#316 with addition of fromcolumns()

* initial work on to/from bcolz petl-developers#310

* resolves petl-developers#310

* added change note

* resolves petl-developers#333 with strict parameter to complement() etc.

* resolves petl-developers#319 with new function groupselectlast

* resolves petl-developers#256 with example in docstring

* doco

* doco

* doco

* Utilize zip_longest to mirror behavior in fromdicts(), filling in None when lengths differ

* make missing value configurable

* fix tests for change in field ordering behaviour

* fix doctest

* fix release notes

* v1.1.0

* fix contributor link

* add field name argument to addrownumbers() to close petl-developers#366

* corrected comments for groupselectmin and groupselectmax

* Fix link in documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment