Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… #24510

faulaire · 2018-12-31T00:00:13Z

Pull request to solve : 24404

closes ValueError when reading a Dataframe with HDFStore in Python 3 from fixed format written in Python 2 #24404
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff

…m fixed format written in Python 2

codecov · 2018-12-31T00:22:19Z

Codecov Report

Merging #24510 into master will not change coverage.
The diff coverage is 0%.

@@           Coverage Diff           @@
##           master   #24510   +/-   ##
=======================================
  Coverage   31.89%   31.89%           
=======================================
  Files         166      166           
  Lines       52421    52421           
=======================================
  Hits        16722    16722           
  Misses      35699    35699

Flag	Coverage Δ
#multiple	`30.29% <0%> (ø)`	⬆️
#single	`31.89% <0%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/pytables.py	`19.53% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 100ffff...66c7e8d. Read the comment docs.

codecov · 2018-12-31T00:22:19Z

Codecov Report

Merging #24510 into master will decrease coverage by 0.01%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master   #24510      +/-   ##
==========================================
- Coverage   31.89%   31.88%   -0.02%     
==========================================
  Files         166      166              
  Lines       52421    52426       +5     
==========================================
- Hits        16722    16717       -5     
- Misses      35699    35709      +10

Flag	Coverage Δ
#multiple	`30.29% <0%> (-0.01%)`	⬇️
#single	`31.88% <0%> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/pytables.py	`19.52% <0%> (-0.01%)`	⬇️
pandas/core/internals/blocks.py	`38.09% <0%> (-0.4%)`	⬇️
pandas/core/frame.py	`27.82% <0%> (-0.14%)`	⬇️
pandas/core/reshape/reshape.py	`7.97% <0%> (-0.09%)`	⬇️
pandas/core/reshape/pivot.py	`8.77% <0%> (ø)`	⬆️
pandas/core/computation/expressions.py	`58.82% <0%> (ø)`	⬆️
pandas/core/generic.py	`31.41% <0%> (ø)`	⬆️
pandas/plotting/_style.py	`23.65% <0%> (ø)`	⬆️
pandas/core/groupby/generic.py	`13.71% <0%> (ø)`	⬆️
pandas/core/arrays/categorical.py	`29.81% <0%> (+0.12%)`	⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 100ffff...173265c. Read the comment docs.

jreback · 2018-12-31T13:39:02Z

so need a sample hdf file that we can include in the repo, generated in py2, that fails on reading (make a new test), and works with your patch. you can put it in here: pandas/tests/io/data/legacy_hdf

read it like

def test_legacy_table_read(self, datapath):
        # legacy table types
        with ensure_clean_store(
                datapath('io', 'data', 'legacy_hdf', 'legacy_table.h5'),
                mode='r') as store:

            with catch_warnings():
                simplefilter("ignore", pd.io.pytables.IncompatibilityWarning)
                store.select('df1')
                store.select('df2')
                store.select('wp1')

jreback

see comments, would also need a whatsnew note

…xed format written in Python 2

jreback · 2018-12-31T16:48:29Z

pandas/io/pytables.py

@@ -2661,6 +2661,8 @@ def read_index_node(self, node, start=None, stop=None):

        if 'name' in node._v_attrs:
            name = _ensure_str(node._v_attrs.name)


you can just directly call this, no if is needed

jreback · 2018-12-31T16:48:45Z

pandas/tests/io/test_pytables.py

+                datapath('io', 'data', 'legacy_hdf', 'legacy_table_fixed_py2.h5'),
+                mode='r') as store:
+            with catch_warnings():
+                simplefilter("ignore", pd.io.pytables.IncompatibilityWarning)


what does this come up?

Copy / past error, this filter is useless

jreback · 2018-12-31T16:49:20Z

doc/source/whatsnew/v0.24.0.rst

@@ -1515,6 +1515,7 @@ MultiIndex
 I/O
 ^^^

+- Fix ``ValueError`` when reading a Dataframe with HDFStore in Python 3 from fixed format written in Python 2


can you add the issue number

Add issue number to whatsnew

Try to decode index name systematically

Remove useless simplefilter

jreback

looks ok, some minor comments, you have some lint issue: https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=6198

jreback · 2018-12-31T17:57:55Z

doc/source/whatsnew/v0.24.0.rst

@@ -1515,6 +1515,7 @@ MultiIndex
 I/O
 ^^^

+- Fix ``ValueError`` when reading a Dataframe with HDFStore in Python 3 from fixed format written in Python 2 (:issue:`24510`)


can you move below (e.g. add it right before Plotting)

Sorry but I didn't understand your request, I'm not familiar with rst format

Ok, I got It, I moved my entry in the file

jreback · 2018-12-31T17:58:56Z

pandas/tests/io/test_pytables.py

@@ -4540,6 +4540,18 @@ def test_pytables_native2_read(self, datapath):
            d1 = store['detector']
            assert isinstance(d1, DataFrame)

+    def test_legacy_table_fixed_format_read_py2(self, datapath):
+        # legacy table with fixed format written en Python 2


add the issue number here as a comments

Change whatsnew category

jreback · 2018-12-31T23:22:20Z

pandas/tests/io/test_pytables.py

+                datapath('io', 'data', 'legacy_hdf',
+                         'legacy_table_fixed_py2.h5'),
+                mode='r') as store:
+            with catch_warnings():


what are the warnings from?

I just used test_legacy_table_read(...) as a template to read the file. I removed this context manager locally and the test is still ok. I'll commit it.

jreback · 2018-12-31T23:35:13Z

lgtm @faulaire ping on green.

jreback · 2019-01-01T01:12:22Z

thanks @faulaire

faulaire · 2019-01-01T13:00:05Z

Thanks for the review & commit @jreback

pandas-dev#24510)

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro…

66c7e8d

…m fixed format written in Python 2

gfyoung added IO HDF5 read_hdf, HDFStore 2/3 Compat labels Dec 31, 2018

jreback requested changes Dec 31, 2018

View reviewed changes

ValueError when reading a Dataframe with HDFStore in Python 3 from fi…

5fff786

…xed format written in Python 2

jreback requested changes Dec 31, 2018

View reviewed changes

faulaire added 3 commits December 31, 2018 17:54

Update v0.24.0.rst

e80ea1b

Add issue number to whatsnew

Update pytables.py

0a7e2e4

Try to decode index name systematically

Update test_pytables.py

4f224f2

Remove useless simplefilter

jreback requested changes Dec 31, 2018

View reviewed changes

faulaire added 2 commits December 31, 2018 19:05

Fix flake8 error on test_pytables.py

554d0c5

Add GH issue number as comment in tests

a33fb2c

Change whatsnew category

jreback requested changes Dec 31, 2018

View reviewed changes

Remove useless catch_warnings context manager from test

173265c

jreback added this to the 0.24.0 milestone Dec 31, 2018

jreback approved these changes Dec 31, 2018

View reviewed changes

jreback merged commit d5283ea into pandas-dev:master Jan 1, 2019

rbenes mentioned this pull request Jan 25, 2019

reading of old pandas dataframe (created in python 2) failed with 0.23.4 #24925

Closed

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… (

cda6c48

pandas-dev#24510)

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… (

5b244b8

pandas-dev#24510)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… #24510

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… #24510

faulaire commented Dec 31, 2018

codecov bot commented Dec 31, 2018

codecov bot commented Dec 31, 2018 •

edited

Loading

jreback commented Dec 31, 2018

jreback left a comment

jreback Dec 31, 2018

faulaire Dec 31, 2018

jreback Dec 31, 2018

faulaire Dec 31, 2018

faulaire Dec 31, 2018

jreback Dec 31, 2018

faulaire Dec 31, 2018

jreback left a comment

jreback Dec 31, 2018

faulaire Dec 31, 2018

faulaire Dec 31, 2018

jreback Dec 31, 2018

faulaire Dec 31, 2018

jreback Dec 31, 2018

faulaire Dec 31, 2018

jreback Dec 31, 2018

jreback commented Dec 31, 2018

jreback commented Jan 1, 2019

faulaire commented Jan 1, 2019

		@@ -2661,6 +2661,8 @@ def read_index_node(self, node, start=None, stop=None):

		if 'name' in node._v_attrs:
		name = _ensure_str(node._v_attrs.name)

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… #24510

Fix ValueError when reading a Dataframe with HDFStore in Python 3 fro… #24510

Conversation

faulaire commented Dec 31, 2018

codecov bot commented Dec 31, 2018

Codecov Report

codecov bot commented Dec 31, 2018 • edited Loading

Codecov Report

jreback commented Dec 31, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Dec 31, 2018

jreback commented Jan 1, 2019

faulaire commented Jan 1, 2019

codecov bot commented Dec 31, 2018 •

edited

Loading