Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feat/data frame methods #121

Merged
merged 327 commits into from
Oct 7, 2020
Merged

Feat/data frame methods #121

merged 327 commits into from
Oct 7, 2020

Conversation

tools4origins
Copy link
Collaborator

@tools4origins tools4origins commented Oct 4, 2020 •
edited

This PR implements most DataFrame methods! 🥳

modified: pysparkling/sql/dataframe.py (Implement most of its methods)
modified: pysparkling/sql/internals.py (Add a few methods that needed #120)
modified: pysparkling/rdd.py (Implement RDD.toDF)

Also, in the following files, it activates tests & logic that use previously not implemented components (mostly by uncommenting said tests/logic):
modified: pysparkling/sql/column.py
modified: pysparkling/sql/expressions/expressions.py
modified: pysparkling/stat_counter.py
modified: pysparkling/tests/test_stat_counter.py
modified: pysparkling/utils.py

This PR is on top of #120 so it also contains its changes

@tools4origins
Implement GetField
4c8b377
@tools4origins
Implement Contains
966f630
@tools4origins
Implement StartsWith
bca996b
@tools4origins
Implement EndsWith
bd6b085
@tools4origins
Implement IsIn
aeb18c3
@tools4origins
Implement IsNotNull
b58c31c
@tools4origins
Implement Cast
d6beaff
@tools4origins
Implement Substring
f7f20ba
@tools4origins
Implement IsNull
d2dd5eb
@tools4origins
Implement Alias
14ac86f
@tools4origins
Expose operator functions
c606a91
@tools4origins
Define SortOrder
9fd0c9c
@tools4origins
Define main orders
acb7b08
@tools4origins
Define Alias orders
5866720
@tools4origins
Implement Literal Expression
d48c464
@tools4origins
Add a format schema util
410be9d
@tools4origins
Add a utils to find an expr position in a schema
93e6f41
@tools4origins
Add a FieldAsExpression function
5c38890
@tools4origins
Implement StarOperator Expression
18fe649
@tools4origins
Implement CaseWhen Expression
0790308
@tools4origins
Implement Otherwise Expression
193d05f
@tools4origins
Implement col function
8b96a83
@tools4origins
Implement when function
4a06d2c
@tools4origins
Add basic Column implementation
3e3ea47
@tools4origins
Implement a column operator parser util
ab68c9c
@tools4origins
Implement Column arithmetic operators
0a3d7a1
@tools4origins
Implement Column arithmetic right operators
a39ccf4
@tools4origins
Implement Column comparison operators
e2a2a15
@tools4origins
Implement Column null safe equal operator
9244178
@tools4origins
Implement Column logic operators
4f36f2e
@tools4origins
Implement DataFrame.intersect
4f1754c
@tools4origins
Implement DataFrame.intersectAll
a6ea851
@tools4origins
Implement DataFrame.subtract
22a6be1
@tools4origins
Implement DataFrame.dropDuplicates
1ebca85
@tools4origins
Implement DataFrame.withColumn
5a050fa
@tools4origins
Implement DataFrame.withColumnRenamed
844ce23
@tools4origins
Implement DataFrame.drop
020be36
@tools4origins
Implement DataFrame.toDF
081c326
@tools4origins
Implement DataFrame.transform
379f581
@tools4origins
Implement a Spark to Panda type converter
754c62f
@tools4origins
Implement DataFrame.toPandas
bbf3742
@tools4origins
Implement DataFrame.drop_duplicates
e8b1b84
@tools4origins
Implement DataFrame.where
9c6eadf
@tools4origins
Implement DataFrame.toPandas
85bb26d
@tools4origins
Enable logic that relies on previously unavailable components
88ed28a
@tools4origins
Enable tests that rely on previously not implemented components
41dbf77
@tools4origins
Fix date to JSON convertion
549acf2
@tools4origins
Clarify XORShiftRandom API
97fe02e
@tools4origins
Align utils with newly implemented functions
40101b9
@tools4origins
Implement RDD.toDF
0f1190c
@tools4origins
Implement DFI sampleBy
02c769a
@tools4origins
Add support of aggregations in DataFrame.select
67ad2c9
@tools4origins
Implement DFI.summary
ee6a05d
@tools4origins
Implement DFI.crosstab
41d4608
@tools4origins
Implement DFI.dropDuplicates
ba84bbc
@tools4origins
Add support of pivot without pivot value being specified
4d6ba0e
@tools4origins
Copy link
Collaborator Author

Travis build is green \o/ :partying_face:

@tools4origins tools4origins mentioned this pull request Oct 4, 2020
Closed
@tools4origins tools4origins merged commit 1217059 into svenkreiss:master Oct 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
1 participant
@tools4origins