New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

contents go missing when loading a generator using .fromdicts() #341

Closed
thatneat opened this Issue Sep 9, 2015 · 3 comments

Comments

Projects
None yet
2 participants
@thatneat
Contributor

thatneat commented Sep 9, 2015

input:

>>> petl.fromdicts(row for row in [{'foo':1,'bar':2},{'foo':3,'bar':4}]).tocsv()

expected output:

bar,foo
1,2
3,4

actual output (only the header survives!):

bar,foo

thatneat added a commit to thatneat/petl that referenced this issue Sep 9, 2015

listify dicts during header inference to fix petl-developers#341
It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header - that's why I didn't just listify the input immediately on line 132.

thatneat added a commit to thatneat/petl that referenced this issue Sep 9, 2015

listify dicts during header inference to fix petl-developers#341
It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys-for-the-header implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header.
@thatneat

This comment has been minimized.

Contributor

thatneat commented Sep 9, 2015

sorry for all the branches; the last one (e2f9f3e) is the one I recommend merging in.

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Sep 11, 2015

Many thanks @thatneat, I'll take a look asap.

On Wednesday, September 9, 2015, thatneat notifications@github.com wrote:

sorry for all the branches; the last one (e2f9f3e
e2f9f3e)
is the one I recommend merging in.


Reply to this email directly or view it on GitHub
#341 (comment).

Alistair Miles
Head of Epidemiological Informatics
Centre for Genomics and Global Health http://cggh.org
The Wellcome Trust Centre for Human Genetics
Roosevelt Drive
Oxford
OX3 7BN
United Kingdom
Web: http://purl.org/net/aliman
Email: alimanfoo@googlemail.com alimanfoo@gmail.com
Tel: +44 (0)1865 287721

@alimanfoo alimanfoo added this to the v1.1 milestone Nov 25, 2015

@alimanfoo

This comment has been minimized.

Collaborator

alimanfoo commented Dec 3, 2015

Working on this now...

@alimanfoo alimanfoo closed this in e2f9f3e Dec 3, 2015

alimanfoo added a commit that referenced this issue Dec 3, 2015

Merge pull request #342 from thatneat/patch-4
listify incoming iterable of dicts to fix #341

alimanfoo added a commit that referenced this issue Dec 3, 2015

andrewakim added a commit to andrewakim/petl that referenced this issue Jul 13, 2016

Pulling in changes and updating my fork of petl (#1)
* work on PY3 compatibility, working through python 3 test failures, some way to go yet

* remove multirangeaggregate; work on flexible comparisons

* fix errors in joins

* work on csv compatibility, unicode todo

* working on csv compatibility, part-baked

* closer to working csv module

* csv module complete?

* csv all working

* rewrite text module

* various fixes

* all tests passing in py26, py27, py34

* dbtests working

* doco

* added support for write_header in tee(u)csv

* docs reworked; consistent state

* review petl.io.pickle documentation

* petl.io.sqlite3 doctests passing

* petl.io.text doctests; change select behaviour for relaxed comparison

* petl.io.text doctests pass

* petl.io.xml doctests pass

* petl.transform.basics doctests pass

* petl.transform.conversions doctests pass

* petl.transform.dedup doctests pass

* petl.transform.conversions doctests pass

* reorganise project layout

* petl.transform.maps doctests passing

* petl.transform.reductions doctests passing

* petl.transform.regex doctests passing

* petl.transform.reshape doctests passing

* petl.transform.selects doctests passing

* petl.transform.setops doctests passing

* petl.transform.sorts and unpacks doctests passing

* refactor petl.util

* fix tests

* combine fluent and interactive into main petl

* complete integration of fluent and interactive into core

* doco rewrite; doctest rewrite

* all doctests passing

* all tests passing

* integrate petlx.display; rework case study 1; add html support for styling; review docs

* doco review; document v1.0 changes

* remove u... functions

* fix docs after removing u... functions

* rework config and vis

* rerun reprhtml nb

* doctests working again after look changes

* v1.0.0-alpha1

* review documentation; change some parameter names

* doco

* v1.0.0a2

* increment version

* doco

* consistify variable naming

* consistify variable naming

* consistify variable names

* doco

* doco

* refactor dbtests

* fix petl executable

* add contributing doc; fix case study

* doco

* v1.0.0a3

* increment version

* resolves petl-developers#303

* resolves petl-developers#295

* resolves petl-developers#294

* various fixes

* closes petl-developers#205

* closes petl-developers#142; closes petl-developers#293 by merging in changes manually

* port array module from petlx

* port array module complete

* migrate dataframe, xls, xlsx

* port hdf5

* doco

* tox running

* merge sqlite3 into db module; migrate db_create from petlx

* doco

* migrate intervals

* doco

* rename modules with optional dependencies; migrate whoosh module

* documentation

* add validation module; add truncate and width to look(); add truncate to display(); rename some variables

* fix doctests

* v1.0.0a4

* increment version

* doco, prep for beta

* v1.0.0b1

* bump version

* add config.look_width

* windows setup

* test_csv pass windows

* test_csv_unicode pass windows

* test_html pass windows

* test_html_unicode pass windows

* fix html tests for cross-platform line endings

* fix test_sources for windows

* fix test_sqlite3 for windows

* fix test_tees for windows

* fix test_text for windows

* fix test_text_unicode for windows

* add excel test resources

* fix missing resources

* test compatibility fixes

* fix excel tests for windows

* fix sort file-caching to work on windows

* test sort cleanup

* fix tempdir in sort tests

* add logging

* get sort cleanup working on windows

* fix test compatibility for windows

* closes petl-developers#245 by moving import sqlite3 inside function, and trying import of cStringIO and cPickle falling back to import StringIO and pickle under PY2

* doco; add finer-grained exception

* v1.0.0b2

* v1.0.0

* minor doco

* doco

* rerun case study

* add tests for petl-developers#308, experiment with implementation

* mid-work

* fixes petl-developers#308

* doco

* v1.0.1

* logging

* add pypy support

* closes petl-developers#309

* v1.0.2

* Catches ValueError thrown when sys.stdin is a pseudofile

When using py.test it redirect stdin/stdout in pseudofiles by
default. Pseudofile do not provide a fileno() method and as such
raise a `ValueError` as soon as `petl.io.sources` is imported (as shown
below).

```
tests/transform/test_transforms.py:1: in <module>
    import petl
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/__init__.py:9: in <module>
    from petl import util
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/__init__.py:13: in <module>
    from petl.util.vis import look, lookall, lookstr, lookallstr, see
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/vis.py:12: in <module>
    from petl.io.sources import MemorySource
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/__init__.py:3: in <module>
    from petl.io.sources import FileSource, GzipSource, BZ2Source, ZipSource, \
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/sources.py:150: in <module>
    stdin_binary = _get_stdin_binary()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/sources.py:133: in _get_stdin_binary
    fd = sys.stdin.fileno()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/_pytest/capture.py:441: in fileno
    raise ValueError("redirected Stdin is pseudofile, has no fileno()")
E   ValueError: redirected Stdin is pseudofile, has no fileno()
```

Simply adding `ValueError` to the tuple of caught exceptions resolves the issues.

* v1.0.3

* doco

* v1.0.4

* closes petl-developers#318

* v1.0.5

* doco

* Use lxml if available to enable parent traversal in xml source

* closes petl-developers#315

* micro version bump

* PEP8; doco

* ack

* doco

* doco

* doco

* doco

* resolve conflict example

* example

* fixes petl-developers#324

* v1.0.7

* Adds support for named paramstyle

* Fixes petl.tocsv() exporting of 'None' string
Petl tocsv exports 'None' instad of no value if used with python 2.x

* add tox

* fix xlsx tests for recent version of openpyxl

* pin versions for optional requirements

* pin sphinx version for rtfd

* petl-developers#326 add documentation note on how to work around cx_Oracle limitation

* micro version bump

* add contributors

* closes petl-developers#325 with tests and fix

* v1.0.9

* closes petl-developers#317 by removing IF EXISTS and trailing semicolon

* v1.0.10

* fix petl-developers#330

* test for petl-developers#330. close petl-developers#330

* Adds test for commit 660bc12
Test `test_tocsv_none()` is added that validates whether `tocsv()` with `None` elements generates empty CSV values.

* Added unit test to cover renaming multiple headers by index

* closes petl-developers#335 with minor changes after PR

* ack

* v1.0.11

* add "strict" keyword doc for rename() function

* ack psnj

* fix for new tox

* listify incoming iterable of dicts to fix petl-developers#341

It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys-for-the-header implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header.

* Per email from Brad Maggard on 2015-09-16, addfieldusingcontext() doesn't have access to the new field, making calculations like running sum more complicated.

The generator yields the tuple from the original table.  It now creates a new Record() object with the additional field and the previous row's value, providing access to the prior row's new field.

* upgrade requirements

* fix doctests for numpy upgrade

* fix for petl-developers#350

* following on from petl-developers#350, ensure all handling of field names is via text_type

* some work on petl-developers#349

* fix petl-developers#347 in general by changing default value for errors kwarg

* close petl-developers#349

* newline

* work on petl-developers#345

* work on petl-developers#345

* work on petl-developers#345

* fixes petl-developers#345

* extension to original fix for petl-developers#341 to avoid complete listification

* fixes petl-developers#209

* fixes petl-developers#332; also adds sortheader() function

* fixes petl-developers#339 by adding biselect

* add sortheader() as Table method

* fix tox for platform differences

* resolves petl-developers#337 by adding epilogue argument

* remove test requirements for pypy

* resolves petl-developers#327 via addition of stack() method; also revisit cat()

* doco; missing method

* resolves petl-developers#323 with documentation and extra test

* doco

* resolves petl-developers#316 with addition of fromcolumns()

* initial work on to/from bcolz petl-developers#310

* resolves petl-developers#310

* added change note

* resolves petl-developers#333 with strict parameter to complement() etc.

* resolves petl-developers#319 with new function groupselectlast

* resolves petl-developers#256 with example in docstring

* doco

* doco

* doco

* Utilize zip_longest to mirror behavior in fromdicts(), filling in None when lengths differ

* make missing value configurable

* fix tests for change in field ordering behaviour

* fix doctest

* fix release notes

* v1.1.0

* fix contributor link

* add field name argument to addrownumbers() to close petl-developers#366

* corrected comments for groupselectmin and groupselectmax

* Fix link in documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment