Skip to content

/orange3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pandas forward compat #1828

Closed
wants to merge 21 commits into from

Conversation

Reviewers

@astaric astaric

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@kernc @codecov-io @astaric @sstanovnik
@kernc
Copy link
Member

commented Dec 16, 2016 •
edited

Description of changes

Minimal (non-exhaustive) pandas forward-compatible API with our current Table technology. Current API still (mostly) works but is deprecated and updated in tests.

Includes
  • Code changes
  • Tests
  • Documentation

@kernc kernc force-pushed the kernc:pandas branch 9 times, most recently from 4ea0ab7 to a572b5d Dec 20, 2016

@kernc kernc changed the title [WIP] Pandas forward compat Pandas forward compat Dec 20, 2016

@kernc kernc assigned astaric Dec 20, 2016

@codecov-io

This comment has been minimized.

Sign in to view
Copy link

commented Dec 20, 2016 •
edited

Current coverage is 89.08% (diff: 85.00%)

Merging #1828 into master will decrease coverage by 0.15%

@@             master      #1828   diff @@
==========================================
  Files            86         85     -1   
  Lines          9100       9169    +69   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           8121       8168    +47   
- Misses          979       1001    +22   
  Partials          0          0

Sunburst

Powered by Codecov. Last update 7acb5fc...a572b5d
@astaric

astaric reviewed Dec 21, 2016
View changes

Orange/misc/distmatrix.py Outdated
@@ -261,10 +262,10 @@ def save(self, filename):
with open(filename, "wt") as fle:
fle.write(data + "\n")
if col_labels is not None:
fle.write("\t".join(str(e.metas[0]) for e in col_labels) + "\n")
fle.write("\t".join(str(m[0]) for m in col_labels.metas) + "\n")

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016

Member

why?
Orange/misc/distmatrix.py Outdated
for i, row in enumerate(self):
if row_labels is not None:
fle.write(str(row_labels[i].metas[0]) + "\t")
fle.write(str(row_labels.metas.T[0][i]) + "\t")

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016

Member

why?
Orange/preprocess/impute.py Outdated
@@ -81,8 +81,7 @@ class DropInstances(BaseImputeMethod):
description = ""

def __call__(self, data, variable):
index = data.domain.index(variable)
return numpy.isnan(data[:, index]).reshape(-1)
return numpy.isnan([list(row) for _, row in data[variable].iterrows()]).reshape(-1)

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016

Member

slow?

This comment has been minimized.

Sign in to view
Copy link
@kernc

kernc Dec 21, 2016

Author Member

Haven't measured it, but I suppose not much slower than it was before. numpy.isnan(data[:, index]) also did a full iteration over data (via Sequence.__iter__() via table.__getitem__(i)).

In pandas, this will be: return data[variable].isnull().

I'll fix it up to use Table.get_column_view() interim.
Orange/statistics/contingency.py Outdated


def get_contingencies(dat, skipDiscrete=False, skipContinuous=False):
def get_contingencies(dat, skip_discrete=False, skip_continuous=False):

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016 •
edited

Member

was this rename really necessary?
Orange/tests/test_contingency.py
@@ -39,8 +39,8 @@ def test_discrete(self):

def test_discrete_missing(self):
d = data.Table("zoo")
d.Y[25] = float("nan")
d[0][0] = float("nan")
d.loc[d.index[25], d.domain.class_var] = float("nan")

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016

Member

I just love how pandas makes the code cleaner :)

This comment has been minimized.

Sign in to view
Copy link
@kernc

kernc Dec 21, 2016 •
edited

Author Member

This is not code. These are tests, made compatible with as little change as possible. The useful function code will always be much cleaner.

Ideally, this would be:

d.type.iloc[25] = np.nan  # 'type' is classvar name
# or
d.target.iloc[25] = np.nan  # target as an introduced alias
# or
d[d.domain.class_var].iloc[25] = np.nan  # familiar, verbose

But our current code makes it hard to adapt it so. I suppose it will have to get a bit more ugly before it gets neat.
Orange/widgets/data/owfeatureconstructor.py
@@ -544,8 +544,7 @@ def duplicateFeature(self):
def check_attrs_values(self, attr, data):
for i in range(len(data)):
for var in attr:
if not math.isnan(data[i, var]) \
and int(data[i, var]) >= len(var.values):
if data[var].iloc[data.index[i]] not in range(var.values):

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016 •
edited

Member

I did not know that you can use "in" with generators :)

This comment has been minimized.

Sign in to view
Copy link
@kernc

kernc Jan 4, 2017 •
edited

Author Member

In Python 3, range(x) is a special object. You can't use in with generators.
Orange/widgets/evaluate/owpredictions.py Outdated
@@ -278,11 +278,6 @@ def _update_predictions_model(self):
results = []
for p in slots:
values, prob = p.results
if p.predictor.domain.class_var.is_discrete:

This comment has been minimized.

Sign in to view
Copy link
@astaric

astaric Dec 21, 2016

Member

why?

@kernc kernc force-pushed the kernc:pandas branch from a572b5d to 755aba8 Jan 5, 2017

sstanovnik added some commits Jul 11, 2016

@sstanovnik @kernc
Some basic fixes for subscripting pandas.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
1618db6
@sstanovnik @kernc
Ported contingency to pandas.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
4399dc1
@sstanovnik @kernc
Adapt distances and tests to work with the new Table. 
Sparse support pending.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
f197839
@sstanovnik @kernc
Miscellaneous test adaptations.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
a996cca
@sstanovnik @kernc
Miscellaneous test compatibility fixes.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
9065ea8
@sstanovnik @kernc
Migrate remover and its tests.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
72cb35e
@sstanovnik @kernc
Simple tree and softmax adaptation. 
Of note is the forced value type to float: without it, integer arrays
are interpreted as double pointers and their values are 0, which breaks
weights.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
28c6548
@sstanovnik @kernc
Use 0 instead of NA when values don't exist in distributions.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
9acdd5e
@sstanovnik @kernc
A bucketload of fixes for widgets. 
Over 90 % of base widgets now work. I haven't checked every single
button, but they produce the intended result. Merge data, feature
constructor and Venn diagram need a bit more work, so those aren't
functional yet.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
6f519ba
@sstanovnik @kernc
Use proper top-level imports. D'oh!
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
8f14ecd
@sstanovnik @kernc
A small fix for the new single-class test.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
e9aeacd
@sstanovnik @kernc
Fix some broken Table imports.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
5eb9004
@sstanovnik @kernc
Widget test adaptation and widget fixes.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
4ab5d18
@sstanovnik @kernc
REVIEWME: 'fixed' displaying SQL tables.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
0dae51d
@sstanovnik @kernc
Fix OWHeatmap and its recent tests.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
fdc7dc5
@sstanovnik @kernc
Remove Table.append.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
3bd4c51

sstanovnik and others added some commits Aug 16, 2016

@sstanovnik @kernc
Change usages of checksum to hash.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
886b5cc
@sstanovnik @kernc
Fix a failing owkmeans test.
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
6e6c547
@kernc
Table, Instance: minimal forward-pandas-compatible API
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
9a1a35a

@kernc kernc force-pushed the kernc:pandas branch from 755aba8 to efa392d Jan 5, 2017

kernc added some commits Dec 15, 2016

@kernc
Avoid Table API deprecations 
Replaced were:

    if [data|table]:                 (for __bool__)
    (data|table)\[                   (changed to pandas indexing)
    \[.{10}(\.\.\.|Ellipsis).{10}\]  (unsupported Ellipsis replaced with
                                      slice(None))
    for row in data                  (replaced with .iterrows() call)
    ...
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
7fad5c7
@kernc
Avoid Table API deprecations (SQL tests)
Verified
This commit was signed with a verified signature.
@kernc kernc
GPG key ID: E0955D0CB6D501E3 Learn about signing commits
Loading status checks…
36a2dea

@kernc kernc force-pushed the kernc:pandas branch from efa392d to 36a2dea Jan 5, 2017

@astaric astaric removed their assignment Apr 7, 2017

@astaric

This comment has been minimized.

Sign in to view
Copy link
Member

commented Dec 20, 2017

As this PR touches too many files in the repository and has not been updated for almost a year, it is highly unlikely that it will be merged. We are still interested in porting the Table to pandas, but it will have to be done in a more gradual way.

If anyone is interested in working on this, I would suggest the following steps:

  1. Add new methods to Table object that match pandas interface
    (empty(), iterrows(), ... - see commit 9a1a35a for more ideas)
  2. Modify the code to call the new methods while deprecating the old ones
    (code calls empty(), __bool__ becomes deprecated, ...)
  3. Figure out if Table can be ported to pandas without modifying more than 10 files, if not, return to 1. :)

Each "pandas compatible" method should be added in a separate pull request, as that eases reviewing and raises the chances of PR actually being merged. I would also split 1. and 2. into two PRs as 1 should only touch a single file and rebasing it should be much easier than the modifications of all places the code is used in 2.

@astaric astaric closed this Dec 20, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.