Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature interval identfier #2

Merged
merged 41 commits into from Mar 16, 2019
Merged

Feature interval identfier #2

merged 41 commits into from Mar 16, 2019

Conversation

mansenfranzen
Copy link
Owner

@mansenfranzen mansenfranzen commented Mar 15, 2019 •
edited

Add interface and pandas implementations for IntervalIdentifier wrangler:

An interval is defined as a range of values beginning with an opening
marker and ending with a closing marker (e.g. the interval daylight may be
defined as all events/values occurring between sunrise and sunset).

The interval identification wrangler assigns ids to values such that values
belonging to the same interval share the same interval id. For example, all
values of the first daylight interval are assigned with id 1. All values of
the second daylight interval will be assigned with id 2 and so on.

TODO:

  • Update changelog
  • Add cumsum based implementation
  • Unify start/end marker notation across tests and implementations
  • Remove flake8 exclude for test folders
  • Open issue to address possible performance increases with numba.jit and pure numpy

@mansenfranzen
Minor rst adjustment.
4f75af2
@mansenfranzen
Add util.sanitizer module including ensure_tuples helper function.
d32dde8
@mansenfranzen
Add BaseIntervalIdentifier.Allow parameter parsing/conversion in
12ea727 
`__init__` method.
@mansenfranzen
Handle None correctly.
f7de85b
@mansenfranzen
Move IntervalIdentifier into separate interfaces module.
f320858
@mansenfranzen
Move tests for IntervalIdentifier into separate test module.
38cbcf9
@mansenfranzen
Add PandasWrangler providing methods common to all pandas based
37a6826 
wranglers.
@mansenfranzen
Add test data for interval identifier wrangler.
dcee237
@mansenfranzen
Add NaiveIterator for interval identification for pandas engine for
f65cf3f 
testing purposes.
@mansenfranzen
Exclude test modules with imported pytest fixtures because flake8 will
4373963 
always report F401 and F811 due to how pytest fixtures work. Putting all
tests in `conftest.py` is not possible because all wrangler tests would
finally be located there which would soon run into namespace issues.
@mansenfranzen
Refactor tests. Add single interval and spanning interval tests.
8027309
@mansenfranzen
Add target_column_name to IntervalIdentifier interface.
dd29360
@mansenfranzen
Add actual implementation for ǸaiveIterator.
8fdf640
@mansenfranzen
Refactor tests to use pytest.mark.parametrize instead of fixture
5827364 
parametrization to run all test cases against all available algorithms.
In addition, tests include different marker types (string, int, float)
and shuffling.
@mansenfranzen
Fix incorrect type annotation.
5c11582
@mansenfranzen
Add note for groupby columns that they should reference partition keys
78c500a 
for distributed comutation engines. Clarfify shortest valid interval
more precisely.
@mansenfranzen
Add type hints.
fb0cd30
@mansenfranzen
Refine is_valid_begin and is_invalid_begin doc strings.
6f5e592
@mansenfranzen
Add groupby tests for IntervalIdentifier.
7113aaa
@mansenfranzen
Add tests for multiple order/groupby columns, invalid start and end
67c6935 
markers for `IntervalIdentifier`. Rearrange and label test input for
better readability.
@mansenfranzen
Refine doc string for NaiveIterator.
2319933
@mansenfranzen
Rename sort_order to ascending and change string parameter type to
7d8aa28 
bool to comply with pandas and spark convention.
@mansenfranzen
Add tests for sort order and invalid begin/close markers. Fix bug which
bdc216b 
caused wrong result when begin marker was left open and was not closed.
Reorder sequence of test parameters for
`test_pandas_interval_identifier` to have wrangler first and test case
second which increases test output readability.
@mansenfranzen
Fix bug which caused wrong result when begin marker was left open and
54e93eb 
was not closed.
@mansenfranzen
Add VectorizedCumSum implementation for interval identification in
b91e93b 
pandas.
@mansenfranzen
Activate all pandas versions.
dc932a9
@mansenfranzen
Add pandas versions to tox dependencies.
6427276
@mansenfranzen
Add pandas versions to tox dependencies.
183bfea
@mansenfranzen
Test excluding specific environments in TravisCI to remove builds which
86ca01b 
do not have wheels.
@mansenfranzen
Simplify doc strings. Unify naming of start and end markers across tests
a16de5f 
and implementations for interval identifier.
@mansenfranzen
Exclude all python/pandas version combinations from TravisCI which do
2a190fb 
not have wheels on pypi because building from source is time consuming
and would even require additional TravisCI configuration. As a trade
off, combinations of old pandas versions with newer python versions are
dropped in favor of maintainability and speed with Travis CI.
@mansenfranzen mansenfranzen added the new feature Add a new feature label Mar 15, 2019
@mansenfranzen mansenfranzen self-assigned this Mar 15, 2019
@mansenfranzen
Remove exlude in flake8 config which is no longer necessary due to
9655afe 
dropping import of pytest fixtures.
@mansenfranzen
Update changelog in regard to additions of IntervalIdentifier.
f12ff63
@mansenfranzen
Test against all versions.
263aa84
@mansenfranzen
Exclude python 3.7 and pandas 0.22.0 due to missing wheels.
acf4f07
@mansenfranzen
Move commonly used types into separate module.
8e0800b
@mansenfranzen
Remove redundant type Any and use types from util.types.
28d3bdc
@mansenfranzen
Add groupby and sort_values methods to PandasWrangler which are
35670a0 
convenient functions taking care of optional parameters. Add
`PandasSingleNoFit` subclass which can be used as a default for
wranglers without fitting required and only a single dataframe as input
and output. Update types to be used from `util.types`.
@mansenfranzen
Refactor via removing duplicated code while adding
1ac20e7 
`BaseIntervalIdentifier`. Add input and output validation.
@mansenfranzen
Streamline naming convention for wrangler and add tests for unused
ccc65e1 
`order_columns` and `groupby_columns` parameters.
@mansenfranzen
Mark _BaseIntervalIdentifier as private and add NotImplementedError
fc94238 
for `_transform`.
@mansenfranzen mansenfranzen merged commit e4aad01 into master Mar 16, 2019
@mansenfranzen mansenfranzen deleted the feature_interval_identfier branch March 16, 2019 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@mansenfranzen mansenfranzen

Labels
new feature Add a new feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@mansenfranzen