Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Showing Source of Conflict #256

pawl opened this issue May 8, 2014 · 2 comments


Copy link

commented May 8, 2014

I'm using mergeduplicates to combine data from two spreadsheets and show the cells with discrepancies. I want the cells with discrepancies to show "Source1: value1, Source2: value2" rather than the unordered Conflict((value2,value1)).

When a conflict occurs, is there a way to determine the source of the conflict after I've done on the data and performed petl.mergeduplicates?

One possibility might be to use petl.conflicts and add a column showing the source to each table? But, I prefer having the duplicates merged for readability purposes.

alimanfoo added a commit that referenced this issue May 9, 2014

This comment has been minimized.

Copy link

commented May 9, 2014

Hi Paul, thanks for raising this. As it currently stands I don't think there's any direct way to associate the source of the conflict with each conflicting value. However, a workaround as you suggest is to use petl.conflicts. Here's a trivial example:

pawl added a commit to pawl/petl that referenced this issue May 28, 2014

@alimanfoo alimanfoo added this to the v1.1 milestone Nov 25, 2015


This comment has been minimized.

Copy link

commented Dec 8, 2015

Closing as can be worked around, however happy to review revised PR if anyone needs this functionality within mergeduplicates.

@alimanfoo alimanfoo closed this Dec 8, 2015

alimanfoo added a commit that referenced this issue Dec 8, 2015
alimanfoo added a commit that referenced this issue Dec 9, 2015
Merge pull request #363 from alimanfoo/issue_256
resolves #256 with example in docstring
andrewakim added a commit to andrewakim/petl that referenced this issue Jul 13, 2016
Pulling in changes and updating my fork of petl (#1)
* work on PY3 compatibility, working through python 3 test failures, some way to go yet

* remove multirangeaggregate; work on flexible comparisons

* fix errors in joins

* work on csv compatibility, unicode todo

* working on csv compatibility, part-baked

* closer to working csv module

* csv module complete?

* csv all working

* rewrite text module

* various fixes

* all tests passing in py26, py27, py34

* dbtests working

* doco

* added support for write_header in tee(u)csv

* docs reworked; consistent state

* review documentation

* doctests passing

* doctests; change select behaviour for relaxed comparison

* doctests pass

* doctests pass

* petl.transform.basics doctests pass

* petl.transform.conversions doctests pass

* petl.transform.dedup doctests pass

* petl.transform.conversions doctests pass

* reorganise project layout

* petl.transform.maps doctests passing

* petl.transform.reductions doctests passing

* petl.transform.regex doctests passing

* petl.transform.reshape doctests passing

* petl.transform.selects doctests passing

* petl.transform.setops doctests passing

* petl.transform.sorts and unpacks doctests passing

* refactor petl.util

* fix tests

* combine fluent and interactive into main petl

* complete integration of fluent and interactive into core

* doco rewrite; doctest rewrite

* all doctests passing

* all tests passing

* integrate petlx.display; rework case study 1; add html support for styling; review docs

* doco review; document v1.0 changes

* remove u... functions

* fix docs after removing u... functions

* rework config and vis

* rerun reprhtml nb

* doctests working again after look changes

* v1.0.0-alpha1

* review documentation; change some parameter names

* doco

* v1.0.0a2

* increment version

* doco

* consistify variable naming

* consistify variable naming

* consistify variable names

* doco

* doco

* refactor dbtests

* fix petl executable

* add contributing doc; fix case study

* doco

* v1.0.0a3

* increment version

* resolves petl-developers#303

* resolves petl-developers#295

* resolves petl-developers#294

* various fixes

* closes petl-developers#205

* closes petl-developers#142; closes petl-developers#293 by merging in changes manually

* port array module from petlx

* port array module complete

* migrate dataframe, xls, xlsx

* port hdf5

* doco

* tox running

* merge sqlite3 into db module; migrate db_create from petlx

* doco

* migrate intervals

* doco

* rename modules with optional dependencies; migrate whoosh module

* documentation

* add validation module; add truncate and width to look(); add truncate to display(); rename some variables

* fix doctests

* v1.0.0a4

* increment version

* doco, prep for beta

* v1.0.0b1

* bump version

* add config.look_width

* windows setup

* test_csv pass windows

* test_csv_unicode pass windows

* test_html pass windows

* test_html_unicode pass windows

* fix html tests for cross-platform line endings

* fix test_sources for windows

* fix test_sqlite3 for windows

* fix test_tees for windows

* fix test_text for windows

* fix test_text_unicode for windows

* add excel test resources

* fix missing resources

* test compatibility fixes

* fix excel tests for windows

* fix sort file-caching to work on windows

* test sort cleanup

* fix tempdir in sort tests

* add logging

* get sort cleanup working on windows

* fix test compatibility for windows

* closes petl-developers#245 by moving import sqlite3 inside function, and trying import of cStringIO and cPickle falling back to import StringIO and pickle under PY2

* doco; add finer-grained exception

* v1.0.0b2

* v1.0.0

* minor doco

* doco

* rerun case study

* add tests for petl-developers#308, experiment with implementation

* mid-work

* fixes petl-developers#308

* doco

* v1.0.1

* logging

* add pypy support

* closes petl-developers#309

* v1.0.2

* Catches ValueError thrown when sys.stdin is a pseudofile

When using py.test it redirect stdin/stdout in pseudofiles by
default. Pseudofile do not provide a fileno() method and as such
raise a `ValueError` as soon as `` is imported (as shown

tests/transform/ in <module>
    import petl
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/ in <module>
    from petl import util
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/ in <module>
    from petl.util.vis import look, lookall, lookstr, lookallstr, see
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/util/ in <module>
    from import MemorySource
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in <module>
    from import FileSource, GzipSource, BZ2Source, ZipSource, \
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in <module>
    stdin_binary = _get_stdin_binary()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/petl/io/ in _get_stdin_binary
    fd = sys.stdin.fileno()
../../.virtualenvs/ernie/local/lib/python2.7/site-packages/_pytest/ in fileno
    raise ValueError("redirected Stdin is pseudofile, has no fileno()")
E   ValueError: redirected Stdin is pseudofile, has no fileno()

Simply adding `ValueError` to the tuple of caught exceptions resolves the issues.

* v1.0.3

* doco

* v1.0.4

* closes petl-developers#318

* v1.0.5

* doco

* Use lxml if available to enable parent traversal in xml source

* closes petl-developers#315

* micro version bump

* PEP8; doco

* ack

* doco

* doco

* doco

* doco

* resolve conflict example

* example

* fixes petl-developers#324

* v1.0.7

* Adds support for named paramstyle

* Fixes petl.tocsv() exporting of 'None' string
Petl tocsv exports 'None' instad of no value if used with python 2.x

* add tox

* fix xlsx tests for recent version of openpyxl

* pin versions for optional requirements

* pin sphinx version for rtfd

* petl-developers#326 add documentation note on how to work around cx_Oracle limitation

* micro version bump

* add contributors

* closes petl-developers#325 with tests and fix

* v1.0.9

* closes petl-developers#317 by removing IF EXISTS and trailing semicolon

* v1.0.10

* fix petl-developers#330

* test for petl-developers#330. close petl-developers#330

* Adds test for commit 660bc12
Test `test_tocsv_none()` is added that validates whether `tocsv()` with `None` elements generates empty CSV values.

* Added unit test to cover renaming multiple headers by index

* closes petl-developers#335 with minor changes after PR

* ack

* v1.0.11

* add "strict" keyword doc for rename() function

* ack psnj

* fix for new tox

* listify incoming iterable of dicts to fix petl-developers#341

It's unfortunate to have to listify an incoming generator, but with the current gather-all-keys-for-the-header implementation, the input has to be fully iterated once.
Generator efficiency is still maintained if you pass in a header.

* Per email from Brad Maggard on 2015-09-16, addfieldusingcontext() doesn't have access to the new field, making calculations like running sum more complicated.

The generator yields the tuple from the original table.  It now creates a new Record() object with the additional field and the previous row's value, providing access to the prior row's new field.

* upgrade requirements

* fix doctests for numpy upgrade

* fix for petl-developers#350

* following on from petl-developers#350, ensure all handling of field names is via text_type

* some work on petl-developers#349

* fix petl-developers#347 in general by changing default value for errors kwarg

* close petl-developers#349

* newline

* work on petl-developers#345

* work on petl-developers#345

* work on petl-developers#345

* fixes petl-developers#345

* extension to original fix for petl-developers#341 to avoid complete listification

* fixes petl-developers#209

* fixes petl-developers#332; also adds sortheader() function

* fixes petl-developers#339 by adding biselect

* add sortheader() as Table method

* fix tox for platform differences

* resolves petl-developers#337 by adding epilogue argument

* remove test requirements for pypy

* resolves petl-developers#327 via addition of stack() method; also revisit cat()

* doco; missing method

* resolves petl-developers#323 with documentation and extra test

* doco

* resolves petl-developers#316 with addition of fromcolumns()

* initial work on to/from bcolz petl-developers#310

* resolves petl-developers#310

* added change note

* resolves petl-developers#333 with strict parameter to complement() etc.

* resolves petl-developers#319 with new function groupselectlast

* resolves petl-developers#256 with example in docstring

* doco

* doco

* doco

* Utilize zip_longest to mirror behavior in fromdicts(), filling in None when lengths differ

* make missing value configurable

* fix tests for change in field ordering behaviour

* fix doctest

* fix release notes

* v1.1.0

* fix contributor link

* add field name argument to addrownumbers() to close petl-developers#366

* corrected comments for groupselectmin and groupselectmax

* Fix link in documentation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
2 participants
You can’t perform that action at this time.