Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changed xrange to range #30

Conversation

earthgecko
Copy link
Contributor

For #8

This fixes the use of xrange function in tests/test_feature_significance.py

xrange has been removed in Python 3 and range in Python 3 is now the same as
xrange in Python 2.7.x (with a caveat that there is evidence that certain
Python 3 range usages are NOT a fast as Python 2.7.x xrange).

Searching the code xrange is only used in tests/test_feature_significance.py,
however that said in the master version there are only 2 calls to xrange and in
the i8_add_python3_support branch version there were 8, so @MaxBenChrist I am
not certain if there was a specific intention there.

All tests pass on Python 3.5.2, without handling the builtins in Python 2.7.x it
is not possible to say that if these tests pass on 2.7.x, it is a small change
and range works as xrange only slower, seeing as very large timeseries data
could be handled, it seems pertainent just to flag it up.

(tsfresh-py352) gary@mc11:/opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh$ ../../bin/python3.5 -m pytest tests/test_feature_significance.py
================================================================= test session starts =================================================================
platform linux -- Python 3.5.2, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 -- /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh/../../bin/python3.5
cachedir: .cache
rootdir: /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh, inifile: setup.cfg
plugins: xdist-1.15.0, cov-2.4.0
[gw0] linux Python 3.5.2 cwd: /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh
[gw1] linux Python 3.5.2 cwd: /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh
[gw2] linux Python 3.5.2 cwd: /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh
[gw3] linux Python 3.5.2 cwd: /opt/python_virtualenv/projects/tsfresh-py352/apps/tsfresh
[gw0] Python 3.5.2 (default, Jul 11 2016, 13:20:59)  -- [GCC 4.8.4]
[gw2] Python 3.5.2 (default, Jul 11 2016, 13:20:59)  -- [GCC 4.8.4]
[gw1] Python 3.5.2 (default, Jul 11 2016, 13:20:59)  -- [GCC 4.8.4]
[gw3] Python 3.5.2 (default, Jul 11 2016, 13:20:59)  -- [GCC 4.8.4]
gw0 [14] / gw1 [14] / gw2 [14] / gw3 [14]
scheduling tests via LoadScheduling

tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_bad
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_binary_features
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_bad
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binomial_target_realvalued_features
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_good
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_mixed_case
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binomial_target_realvalued_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_binary_features
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_bad
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_bad
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_mixed_case
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_binary_features
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binomial_target_realvalued_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_mixed_case
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_binary_features
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_mixed_case
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_mixed_case
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binomial_target_realvalued_features Coverage.py warning: No data was collected.


----------- coverage: platform linux, python 3.5.2-final-0 -----------
Name                                                 Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------------------------
tsfresh/convenience/relevant_extraction.py              17     11      6      0    26%   43-60
tsfresh/examples/__init__.py                             2      2      0      0     0%   6-7
tsfresh/examples/robot_execution_failures.py            46     46     16      0     0%   5-121
tsfresh/feature_extraction/extraction.py                41     32     26      0    13%   67-101, 165-190
tsfresh/feature_extraction/feature_calculators.py      258    149     72      1    33%   56-60, 74-77, 105, 129, 149, 163, 177, 191, 204, 220, 242-250, 268-273, 291-292, 311, 329, 348-349, 362, 375, 389, 403, 417, 432-433, 448-449, 467, 482, 497, 512, 527, 541-542, 556-557, 571-572, 586-587, 608-618, 647-651, 670-675, 693, 721-746, 768-782, 808-833, 853-869, 906-913, 935-937, 953-954, 970-971, 984, 997, 1013-1016, 1034, 87->89
tsfresh/feature_extraction/settings.py                 116     96     87      0    10%   67-72, 81-107, 121-127, 148-198, 214-230, 244-278, 291-312
tsfresh/feature_selection/__init__.py                    2      0      0      0   100%
tsfresh/feature_selection/feature_selector.py           66      6     28      2    89%   88, 151-158, 87->88, 149->151
tsfresh/feature_selection/selection.py                  21     13     10      0    26%   85-102
tsfresh/feature_selection/settings.py                   11      0      0      0   100%
tsfresh/feature_selection/significance_tests.py         62     12     16      5    71%   121-126, 206, 208, 225-228, 245-248, 116->121, 205->206, 207->208, 224->225, 244->245
tsfresh/transformers/feature_augmenter.py               23     23      2      0     0%   5-140
tsfresh/transformers/feature_selector.py                21     21      8      0     0%   5-105
tsfresh/transformers/relevant_feature_augmenter.py      46     46     14      0     0%   5-238
tsfresh/utilities/dataframe_functions.py               103     91     86      0     6%   30-35, 57-58, 71-72, 100-134, 145-160, 178-186, 228-290
------------------------------------------------------------------------------------------------
TOTAL                                                  835    548    371      8    26%


============================================================== 14 passed in 5.27 seconds ==============================================================

@coveralls
Copy link

Coverage Status

Coverage remained the same at 94.85% when pulling 42f0a8a on earthgecko:i8_add_python3_support_earthgecko into 3b2c204 on blue-yonder:i8_add_python3_support.

@MaxBenChrist
Copy link
Collaborator

Thank you for your explanation and the commit!

But, the master also contains 8 calls to xrange in test_feature_significance.py, see

https://github.com/blue-yonder/tsfresh/blob/f20ecf07ab221cfc99a1740f751c33617105c89f/tests/test_feature_significance.py

There was no specific intention to use xrange. Another colleague wrote the tests with a preference for xrange (We try to let different people write tests and code)

@earthgecko earthgecko mentioned this pull request Nov 2, 2016
@earthgecko
Copy link
Contributor Author

Handling builtins

Related to #8 (comment)

Getting the codebase and tests aligned to run on both Python 2.7.x and
Python 3.5.x is going to require quite a bit of bifurcation so that imports are
handled correctly. For example:

try:
  from builtins import zip
except ImportError:
  from future_builtins import zip

However this would have ramifications on the zip function possibly as per
https://docs.python.org/2/library/future_builtins.html

The use of builtins as it stands now in i8_add_python3_support is as follows:

tsfresh/tsfresh/feature_extraction/extraction.py - 1 builtins imports found
from builtins import str
tsfresh/tsfresh/feature_extraction/feature_calculators.py - 1 builtins imports found
from builtins import range
tsfresh/tsfresh/feature_extraction/settings.py - 4 builtins imports found
from builtins import zip
from builtins import str
from builtins import range
from builtins import object
tsfresh/tsfresh/examples/robot_execution_failures.py - 1 builtins imports found
from builtins import map
tsfresh/tsfresh/feature_selection/significance_tests.py - 1 builtins imports found
from builtins import str
tsfresh/tsfresh/feature_selection/feature_selector.py - 3 builtins imports found
from builtins import zip
from builtins import range
tsfresh/tsfresh/feature_selection/settings.py - 1 builtins imports found
from builtins import object
tsfresh/tests/transformers/test_feature_selector.py - 1 builtins imports found
from builtins import range
tsfresh/tests/feature_extraction/test_feature_calculations.py - 1 builtins imports found
from builtins import range

However at the moment would possible to just handle builtins with a try and
an innoccous except, maybe something like e.g.

try:
    from builtins import zip
except ImportError:
    from sys import version_info

Perhaps there is scope to add

from __future__ import absolute_import

As per http://python-future.org/compatible_idioms.html#imports-relative-to-a-package
this would make Py2 code safer (more like Py3) by preventing implicit relative
imports.

I am not certain of your thoughts on this. Porting to Python 3 is a lot of
"little" things, but little things can have bigger implications on design, style
and function.

I am certain this port to Python 3 is going to raise lots of little questions,
which have a number of design and possibly even performance/refactor issues that
may need to be addressed.

Does blue-yonder have any preferred methods?

@MaxBenChrist
Copy link
Collaborator

Thanks for your explanations. I will read into best practices about the importing issues with builtins.

Actually, I do not have much experience with python 3.5.x.

What do you think about the changes I made at the 'blue-yonder:i8_add_python3_support' branch so far?

@MaxBenChrist MaxBenChrist merged commit 47c1ff0 into blue-yonder:i8_add_python3_support Nov 2, 2016
@earthgecko
Copy link
Contributor Author

Substituting all builtins and past.builtin

Substituting all builtins and past.builtin with the following example from
tsfresh/feature_extraction/settings.py, allows all tests/test_feature_significance.py
to pass on python-2.7.12 as well.

try:
    from builtins import zip
    from builtins import str
    from builtins import range
    from past.builtins import basestring
    from builtins import object
except ImportError:
    from sys import version_info

This try and except method was used to replace all builtins in the files listed
in my previous comment.

This has not been added or merged to this pull request, it is just testing a PoC
of how to possibly handle the use of builtins in a backwards compatible manner,
I cannot say for certain that this is a viable or even Pythonic way to handle
this, but it appears to work from the pure tests point of view.

At some point I guess I should play with some timeseries data and the package
itself as I have yet to have done that :) All in good time :)

(tsfresh-py2712) gary@mc11:/opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh$ ../../bin/python2.7 -m pytest tests/test_feature_significance.py
==================================================================== test session starts ====================================================================
platform linux2 -- Python 2.7.12, pytest-3.0.3, py-1.4.31, pluggy-0.4.0 -- /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh/../../bin/python2.7
cachedir: .cache
rootdir: /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh, inifile: setup.cfg
plugins: xdist-1.15.0, cov-2.4.0
[gw0] linux2 Python 2.7.12 cwd: /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh
[gw1] linux2 Python 2.7.12 cwd: /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh
[gw2] linux2 Python 2.7.12 cwd: /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh
[gw3] linux2 Python 2.7.12 cwd: /opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh
[gw0] Python 2.7.12 (default, Aug 20 2016, 09:15:25)  -- [GCC 4.8.4]
[gw1] Python 2.7.12 (default, Aug 20 2016, 09:15:25)  -- [GCC 4.8.4]
[gw2] Python 2.7.12 (default, Aug 20 2016, 09:15:25)  -- [GCC 4.8.4]
[gw3] Python 2.7.12 (default, Aug 20 2016, 09:15:25)  -- [GCC 4.8.4]
gw0 [14] / gw1 [14] / gw2 [14] / gw3 [14]
scheduling tests via LoadScheduling

tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_bad
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_bad
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binomial_target_realvalued_features
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_binary_features
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_binary_features
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binomial_target_realvalued_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_good
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_good
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binomial_target_realvalued_features
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_bad
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_all_features_bad
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binomial_target_realvalued_features
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_binary_target_binary_features
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_binary_features
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_mixed_case
tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_mixed_case
[gw1] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_binary_features
[gw0] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCase::test_real_target_mixed_case
[gw2] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_real_target_mixed_case
[gw3] PASSED tests/test_feature_significance.py::FeatureSignificanceTestCaseWithOtherHypothesis::test_binary_target_mixed_case Coverage.py warning: No data was collected.


---------- coverage: platform linux2, python 2.7.12-final-0 ----------
Name                                                 Stmts   Miss Branch BrPart  Cover   Missing
------------------------------------------------------------------------------------------------
tsfresh/convenience/relevant_extraction.py              17     11      6      0    26%   43-60
tsfresh/examples/__init__.py                             2      2      0      0     0%   6-7
tsfresh/examples/robot_execution_failures.py            46     46     16      0     0%   5-121
tsfresh/feature_extraction/extraction.py                44     32     24      0    18%   71-105, 169-194
tsfresh/feature_extraction/feature_calculators.py      261    149     46      1    37%   60-64, 78-81, 109, 133, 153, 167, 181, 195, 208, 224, 246-254, 272-277, 295-296, 315, 333, 352-353, 366, 379, 393, 407, 421, 436-437, 452-453, 471, 486, 501, 516, 531, 545-546, 560-561, 575-576, 590-591, 612-622, 651-655, 674-679, 697, 725-750, 772-786, 812-837, 857-873, 910-917, 939-941, 957-958, 974-975, 988, 1001, 1017-1020, 1038, 91->93
tsfresh/feature_extraction/settings.py                 119    100     66      0    10%   17-20, 74-79, 88-114, 128-134, 155-205, 221-237, 251-285, 298-319
tsfresh/feature_selection/__init__.py                    2      0      0      0   100%
tsfresh/feature_selection/feature_selector.py           69      7     24      2    88%   19, 93, 156-163, 92->93, 154->156
tsfresh/feature_selection/selection.py                  21     13     10      0    26%   85-102
tsfresh/feature_selection/settings.py                   14      0      0      0   100%
tsfresh/feature_selection/significance_tests.py         65     12     16      5    72%   125-130, 210, 212, 229-232, 249-252, 120->125, 209->210, 211->212, 228->229, 248->249
tsfresh/transformers/feature_augmenter.py               23     23      2      0     0%   5-140
tsfresh/transformers/feature_selector.py                21     21      8      0     0%   5-105
tsfresh/transformers/relevant_feature_augmenter.py      46     46     14      0     0%   5-238
tsfresh/utilities/dataframe_functions.py               103     91     84      0     6%   30-35, 57-58, 71-72, 100-134, 145-160, 178-186, 228-290
------------------------------------------------------------------------------------------------
TOTAL                                                  853    553    316      8    28%


================================================================= 14 passed in 4.60 seconds =================================================================
(tsfresh-py2712) gary@mc11:/opt/python_virtualenv/projects/tsfresh-py2712/apps/tsfresh$

@earthgecko
Copy link
Contributor Author

Hi @MaxBenChrist

Hopefully the above answers your question somewhat in terms of what I think about your changes so far. I have not actually executed any code yet. Been building testing and dev virtualenvs, githubbing and PoC'ing :)

However that said. In my humble opinion I would suggest you start considering merging very small changes, very gradually so that i8_add_python3_support does not drift from master too much. From my limited experience with Python 3.5 although there are very small things, they do really influence design, structure and sometimes even performance.

My gut feel (with very limited experience) is that the addition of Python 3 support may have some more overall ramifications, I reckon there is some probability :)

That said, very small changes to your 2.7 master branch that SLOWLY bring that into line should help to not end up with 2 massively drifted branches.

You people are now BUSY, you just got starred and social coding can be overwhelming. It is an entire new dimension :) And popular can be like an avalanche of things hitting you.

So that is what I think about your changes in general :) I do not envy you :)

I think maybe you should perhaps all take a moment to breathe and consider what you all think is important and what you all think the priorities should be, try not to let the all the social coding issues drive you, but the ones that you all think are important.

After all a lot of things have not gone python 3 due to various constraints in the ecosysytem regarding pypy scipy numpy et al. So how important is 3? :)

That said, have diffed master and i8_add_python3_support, maybe Python 3 support is not that vast amounts of change. I am happy to continue and try and see where it gets.

Learning tsfresh backwards :)

I hope that helps.

MaxBenChrist added a commit that referenced this pull request Nov 4, 2016
* use python3 compatible print function

* use absolute import for py3 support

* in py3 one has to load reduce

* check for basestring in column name

* import range for py3 support

* call list of range iterator

* fixed standard library import

* added builtin support for str, basestring, zip, object

* class FeatureExtractionSettings inherits from baseobject

* call list of iterator objects

* change iter() to iteritems()

* replaced filter by list comprehension

* added future to requirements.txt

* Changed xrange to range for #8 (#30)

* csort range py3 compatible (#32)

* Changed xrange to range for #8

* Modified csort due to range changing from list to type for #8

* Use assertItemsEqual py2 and assertCountEqual py3
Due to a change in unittest in as identified by @jneuff in
#8 (comment)

Semantically they appear to be the same and this fixes the related failing
tests on Python 3.5 as described in the gist in
#8 (comment)

Adds a basic method to determine python version for now, only committing so that
the new deeper unitest.assertEqual issue that now presents itself can be
addressed.

* pinned version of future package

* added six to test-requirements.txt

* use six for assertCountEqual unit testing

* deleted some comments

* use six for assertCountEqual to test feature_augmentor

__dict__ will give iterator for properties

* use six.assertCountEqual

* use assertGreaterEqual to test subset

* parameter have to be sorted in feature name

fixes bug #29, features with multiple, but same parameters could end up
with different names

* added python 3.5.2 to .travis.yml

closes #8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants