Refactor LatLong and Datetime Primitives into Separate Files #1861

jacobboney · 2022-01-24T04:03:36Z

Pull Request Description

Fixes Split Transform Primitives out into separate files #1855

Changes:
I decided to split all classes containing Lat/Long functions into their own file as well as classes containing date/time into their own file. In each file I also organized classes in alphabetical order. I don't believe there are any conflicts with the new files as I was able to run the tests.

Comments:
Whenever someone is able to review my changes I would also appreciate some input/advice on the testing. I am running them as described on Ubuntu. They run to the end but I do have some failed tests, not sure if this is due to my changes or if it is just part of the process.

As an aside I apologize for all of the unnecessary commits. I'm still getting the hang of it and understand now I may have gone overboard. Also, I accidentally deleted my original branch which is why I am submitting a second pull request.

…om lat_long_transform.py

Issue#1855 split primitive 2

…o issue#1855-Split_Primitive

…_primitive; added all classes containing date and/or time functions to date_time_transform_primitive

dvreed77 · 2022-01-24T15:01:43Z

@jacobboney Thank you for the submission. It looks like many tests are breaking, have you run make test locally?

thehomebrewnerd · 2022-01-24T15:03:37Z

@jacobboney Thank you for the submission. It looks like many tests are breaking, have you run make test locally?

@jacobboney @dvreed77 There is an issue with the latest version of pandas. We are going to need to restrict our max pandas version before most of these tests will pass. I'm working on a PR for that now. Once my PR gets merged, @jacobboney will need to update his fork with the latest changes from main. Until then, I expect most of these tests will continue to fail our CI here.

gsheni · 2022-01-24T15:49:16Z

featuretools/primitives/standard/api.py

+from .date_time_transform_primitive import *
+from .lat_long_transform_primitive import *


Can we name this file different?

Suggested change

from .date_time_transform_primitive import *

from .lat_long_transform_primitive import *

from .datetime_transform_primitives import *

from .latlong_transform_primitives import *

Of course, I'll implement those name changes today.

jacobboney · 2022-01-24T17:21:12Z

@dvreed77 @thehomebrewnerd
I am a little new to testing. Make test runs on my machine to completion. Is there a log file or something that I'm able to check to see what is actually failing? As of right now I only have the results displayed on the terminal. It passes the vast majority of the tests and only fails a few locally. I have a screenshot of this, but I don't have access to it right now.

gsheni · 2022-01-24T20:35:35Z

@jacobboney You will need to update your branch to get the latest from main. In addition, you will need to update the release notes. Here is an example of that:

https://github.com/alteryx/featuretools/pull/1862/files#diff-eb36f1b6f2ee2691f609c07d32a8d977e4052a9e47d0ec94a241c6fc001a5d5cR37

…_primitive; added all classes containing date and/or time functions to date_time_transform_primitive

… release notes

jacobboney · 2022-01-25T03:09:07Z

The two tests that fail are:
test_deserialize_features_s3
test_checks_primitives_correct_type

I looked at each test and am unsure what may be causing the failure.

thehomebrewnerd · 2022-01-25T14:03:17Z

The two tests that fail are: test_deserialize_features_s3 test_checks_primitives_correct_type

I looked at each test and am unsure what may be causing the failure.

For the test test_checks_primitives_correct_type, I think that test is failing because the error message that is hard-coded into the test no longer matches the error message that is generated, because the location of the primitive has changed. For that test, I think you just need to update the transform_primitive part of the error message to match the new location: datetime_transform_primitives.

The test_deserialize_features_s3 test is a bit trickier. This test is attempting to recreate features from a JSON file stored on S3. However, because the module names have changed in your code, the module name stored in the file no longer contains the Haversine primitive, so the deserialization fails. To fix this we are going to need to change our serialization schema version as well as upload a new JSON file to S3 to use for this test. Let's tackle this one last, after all the other updates have been made and all open PR comments have been addressed.

jacobboney · 2022-01-25T16:24:39Z

Sounds good, I'll look into the issue with first test then. Thank you for your input.

thehomebrewnerd

Other than one minor comment regarding release notes wording, this looks fine to me. Let me know when you are ready to address the remaining failing serialization test, and I can help with getting that resolved.

thehomebrewnerd · 2022-01-25T17:27:23Z

docs/source/release_notes.rst

        * Add DateToHoliday Transform Primitive (:pr:`1848`)
        * Add DistanceToHoliday Transform Primitive (:pr:`1853`)
        * Temporarily restrict pandas and koalas max versions (:pr:`1863`)
+        * Split date, time, latitude and longitude primitives into separate files (:pr:`1861`)


This is a very minor nitpick, but I'd probably say Split Datetime and LatLong primitives into separate files here to make it a little more clear we are talking about primitives that specifically operate on Datetime or LatLong column inputs. Not a big deal though.

jacobboney · 2022-01-25T18:41:01Z

Will do. I won't be able to get to those initial fixes until later in the evening. If that ends up being to late I will be available all day tomorrow to work on the final fix.

…correct_type

jacobboney · 2022-01-26T04:21:21Z

The test_checks_primitives_correct_type issue seems to be resolved. I ran the tests again and there was no failure.

thehomebrewnerd · 2022-01-26T13:43:26Z

The test_checks_primitives_correct_type issue seems to be resolved. I ran the tests again and there was no failure.

@jacobboney Thanks for the update. Looks like you will need to resolve the merge conflicts in the release notes file before the full CI tests will run again.

jacobboney · 2022-01-26T15:56:17Z

Alright, after some Git confusion on my part I think I've resolved the merge conflicts.

dvreed77 · 2022-01-27T21:09:31Z

@jacobboney. I uploaded a new test JSON file to S3 to allow your PR to pass. Can you make the following changes in your code:

File: featuretools/feature_base/features_serializer.py
Line #: 10
Change: SCHEMA_VERSION = "8.0.0"

File: featuretools/tests/primitive_tests/test_feature_serialization.py
Line #: 46
Change: TEST_FILE = "test_feature_serialization_feature_schema_{}_entityset_schema_{}_2022_01_27.json".format(SCHEMA_VERSION, ENTITYSET_SCHEMA_VERSION)

jacobboney · 2022-01-27T21:26:13Z

@dvreed77 Great, thank you. I'll make those changes this evening.

…ialization test to the proper JSON file

jacobboney · 2022-01-28T03:30:14Z

I was able to make all of the changes, however, after running the tests, the features serializer test still failed. I found that the schema version also had to be updated in test_features_serializer.

Unfortunately, there still seems to be an issue with koalas/pandas. The failures come from:
-featuretools/tests/primitive_tests/test_transform_features.py
-featuretools/tests/synthesis/test_koalas_dfs.py

All failures seem to be assertion errors.

codecov · 2022-01-28T14:01:12Z

Codecov Report

Merging #1861 (32c84c7) into main (ff08a6a) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #1861   +/-   ##
=======================================
  Coverage   98.82%   98.82%           
=======================================
  Files         145      147    +2     
  Lines       16277    16290   +13     
=======================================
+ Hits        16086    16099   +13     
  Misses        191      191

Impacted Files	Coverage Δ
...retools/primitives/standard/transform_primitive.py	`100.00% <ø> (ø)`
...ols/tests/synthesis/test_deep_feature_synthesis.py	`99.33% <ø> (ø)`
featuretools/feature_base/features_serializer.py	`100.00% <100.00%> (ø)`
featuretools/primitives/standard/api.py	`100.00% <100.00%> (ø)`
...imitives/standard/datetime_transform_primitives.py	`100.00% <100.00%> (ø)`
...rimitives/standard/latlong_transform_primitives.py	`100.00% <100.00%> (ø)`
...s/primitive_tests/test_datetoholiday_primitives.py	`100.00% <100.00%> (ø)`
...imitive_tests/test_distancetoholiday_primitives.py	`100.00% <100.00%> (ø)`
...ests/primitive_tests/test_feature_serialization.py	`99.35% <100.00%> (ø)`
.../tests/primitive_tests/test_features_serializer.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ff08a6a...32c84c7. Read the comment docs.

jacobboney · 2022-01-28T14:08:52Z

Well, after seeing that all tests passed I guess that means there must be some version issue on my side. Glad to see there are no more errors though.

thehomebrewnerd · 2022-01-28T14:10:09Z

I was able to make all of the changes, however, after running the tests, the features serializer test still failed. I found that the schema version also had to be updated in test_features_serializer.

Unfortunately, there still seems to be an issue with koalas/pandas. The failures come from: -featuretools/tests/primitive_tests/test_transform_features.py -featuretools/tests/synthesis/test_koalas_dfs.py

All failures seem to be assertion errors.

@jacobboney Not exactly sure what might be happening, but all of our CI tests pass with the latest code you pushed, so I think we are good now, unless the final reviews highlight anything else that needs to be updated.

jacobboney · 2022-01-28T14:22:56Z

@thehomebrewnerd Not a problem at all. I was just looking for a issue in the code that wasn't there. It must be something on my end that I'll have to figure out. I'll just stay tuned then for any possible final changes.

@dvreed77 @thehomebrewnerd @gsheni Thank you all very much for your guidance through this process. I greatly appreciate it.

dvreed77 · 2022-01-28T15:06:58Z

Congrats @jacobboney! Thanks for contribution!

jacobboney · 2022-01-28T15:12:44Z

Great! Thank you!

dvreed77 · 2022-03-07T23:35:38Z

fixes #1855

jacobboney and others added 12 commits January 22, 2022 08:48

Split Lat and Long into separate file

0e09947

added lat_long_transform.py to api and removed uneccessary imports fr…

537884c

…om lat_long_transform.py

Changed name of lat long file

1531e98

Merge pull request #1 from jacobboney/issue#1855-Split_Primitive_2

51ac48a

Issue#1855 split primitive 2

Removed uneccessary import

335b4d8

Merge branch 'main' of https://github.com/jacobboney/featuretools int…

3b3619e

…o issue#1855-Split_Primitive

Update Main

81f4ccf

Merge branch 'main' of https://github.com/jacobboney/featuretools int…

59e2060

…o issue#1855-Split_Primitive

Fixed disjointed files on my end

9ed9764

Fixed too many line error and no ending new line error

a7b301e

Reorganized files

3ef87b3

Added all classes containing lat/long functions to lat_long_transform…

0f36f72

…_primitive; added all classes containing date and/or time functions to date_time_transform_primitive

auto-assign bot assigned jacobboney Jan 24, 2022

dvreed77 changed the title ~~Issue#1855 split primitive~~ Refactor LatLong and Datetime Primitives into Separate Files Jan 24, 2022

gsheni reviewed Jan 24, 2022

View reviewed changes

jacobboney and others added 10 commits January 24, 2022 20:27

Merge branch 'main' into issue#1855-Split_Primitive

077beab

Merge branch 'alteryx:main' into main

9af6f2f

Removed uneccessary import

72fd20f

Fixed disjointed files on my end

60330c5

Fixed too many line error and no ending new line error

6ec3a5f

Reorganized files

bd891d3

Added all classes containing lat/long functions to lat_long_transform…

3f90534

…_primitive; added all classes containing date and/or time functions to date_time_transform_primitive

Updated branch from main, Renamed datetime and latlong files, Updated…

1173454

… release notes

Fixed merging conflicts

2ac3a69

Updated imports for date tests

fde92f7

gsheni requested review from dvreed77 and thehomebrewnerd January 25, 2022 16:01

thehomebrewnerd reviewed Jan 25, 2022

View reviewed changes

Updated Release Notes; fixed error issue with test_checks_primitives_…

93953aa

…correct_type

jacobboney added 3 commits January 26, 2022 06:56

Update Main

da13fb0

Applied Branch changes

3e1b8c4

Merged with main

679aa0c

jacobboney added 3 commits January 27, 2022 17:10

Merge with main; updates to release notes

6510eb7

Updated feature serializer's SCHEMA_VERSION and point the feature ser…

299a200

…ialization test to the proper JSON file

Updated features serializer test to also use updated SCHEMA_VERSION

6856ed3

Update release_notes.rst

32c84c7

dvreed77 approved these changes Jan 28, 2022

View reviewed changes

dvreed77 merged commit ec8a72b into alteryx:main Jan 28, 2022

dvreed77 mentioned this pull request Feb 11, 2022

v1.5.0 #1901

Merged

		from .date_time_transform_primitive import *
		from .lat_long_transform_primitive import *

Refactor LatLong and Datetime Primitives into Separate Files #1861

Refactor LatLong and Datetime Primitives into Separate Files #1861

Uh oh!

Conversation

jacobboney commented Jan 24, 2022 • edited by gsheni Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Description

Uh oh!

dvreed77 commented Jan 24, 2022

Uh oh!

thehomebrewnerd commented Jan 24, 2022

Uh oh!

gsheni Jan 24, 2022

Choose a reason for hiding this comment

Uh oh!

jacobboney Jan 24, 2022

Choose a reason for hiding this comment

Uh oh!

jacobboney commented Jan 24, 2022

Uh oh!

gsheni commented Jan 24, 2022

Uh oh!

jacobboney commented Jan 25, 2022

Uh oh!

thehomebrewnerd commented Jan 25, 2022

Uh oh!

jacobboney commented Jan 25, 2022

Uh oh!

thehomebrewnerd left a comment

Choose a reason for hiding this comment

Uh oh!

thehomebrewnerd Jan 25, 2022

Choose a reason for hiding this comment

Uh oh!

jacobboney commented Jan 25, 2022

Uh oh!

jacobboney commented Jan 26, 2022

Uh oh!

thehomebrewnerd commented Jan 26, 2022

Uh oh!

jacobboney commented Jan 26, 2022

Uh oh!

dvreed77 commented Jan 27, 2022

Uh oh!

jacobboney commented Jan 27, 2022

Uh oh!

jacobboney commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jacobboney commented Jan 28, 2022

Uh oh!

thehomebrewnerd commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobboney commented Jan 28, 2022

Uh oh!

dvreed77 commented Jan 28, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobboney commented Jan 28, 2022

Uh oh!

dvreed77 commented Mar 7, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jacobboney commented Jan 24, 2022 •

edited by gsheni

Loading

jacobboney commented Jan 28, 2022 •

edited

Loading

codecov bot commented Jan 28, 2022 •

edited

Loading

thehomebrewnerd commented Jan 28, 2022 •

edited

Loading

dvreed77 commented Jan 28, 2022 •

edited

Loading