Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(dataFrameShow): Remove extra new lines #163

Open
wants to merge 121 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
ee67e40
Add Antl4r as optional dev dependency
tools4origins Sep 18, 2020
644c341
Add grammar files to repository
tools4origins Sep 18, 2020
ac2f991
Document grammar file status
tools4origins Sep 18, 2020
a7eb397
Translate BRACKETED_COMMENT scala condition in python
tools4origins Sep 18, 2020
c2eb275
Align generated code format with project one
tools4origins Sep 18, 2020
eee4564
Implement AST builder
tools4origins Sep 18, 2020
cefb1a1
Document AST module
tools4origins Sep 18, 2020
517d122
Add AST utils
tools4origins Sep 18, 2020
308396e
Convert AST to Python objects
tools4origins Sep 18, 2020
4e1477f
Add basic unittest on SQL parsing
tools4origins Sep 18, 2020
53ff4ab
Base implementation of binary operation
tools4origins Sep 19, 2020
3ea3b16
Parse unary operations
tools4origins Sep 19, 2020
85c0546
Add missing wrapper
tools4origins Sep 19, 2020
9940076
Add wrappers that should never be encountered (only their children sh…
tools4origins Sep 19, 2020
e2b4cdd
Add parenthesis-like contexts
tools4origins Sep 19, 2020
b313b71
Fix isValidDecimal
tools4origins Sep 19, 2020
a8fffeb
Clarify support status of some statements
tools4origins Sep 19, 2020
57dbe30
Clarify namespace support status
tools4origins Sep 19, 2020
89e71c0
Parse join types
tools4origins Sep 19, 2020
7742cf0
Move correctly implemented parsers
tools4origins Sep 19, 2020
6d460fd
Enhance logging integration in unittests
tools4origins Sep 19, 2020
0fc115b
Fix antlr4 warning
tools4origins Sep 19, 2020
dde6a5d
Make method that can be static static
tools4origins Sep 19, 2020
e24646b
Add a method to convert a SQL string to an AST
tools4origins Sep 19, 2020
7f95491
Fix string parsing
tools4origins Sep 19, 2020
ff4e7bb
Simplify test
tools4origins Sep 19, 2020
6914b50
Use SQL to parse types
tools4origins Sep 19, 2020
8ecd9b8
Add type aliases
tools4origins Sep 19, 2020
330b6dd
Test type parsing
tools4origins Sep 19, 2020
16ef30e
Add parameterized as a dependency
tools4origins Sep 19, 2020
a6932a8
Clarify function name
tools4origins Sep 19, 2020
ef94080
Remove unused check
tools4origins Sep 19, 2020
8572580
Detect array types
tools4origins Sep 19, 2020
83fe840
Detect map types
tools4origins Sep 19, 2020
4d5beff
Detect struct type
tools4origins Sep 19, 2020
fb032eb
Remove duplicated code
tools4origins Sep 19, 2020
5af7c22
Implement SQL struct builder
tools4origins Sep 20, 2020
aafb26a
Implement logical Not
tools4origins Sep 20, 2020
ad570cf
Add a debug mode to sql parsing function
tools4origins Sep 22, 2020
fd9e659
Fix operation lookup when name contains letters
tools4origins Sep 22, 2020
d81073d
Clarify exception class
tools4origins Sep 22, 2020
f0f9523
Add expression parser
tools4origins Sep 22, 2020
569f271
Enhance literal parsing
tools4origins Sep 22, 2020
2cdc6c9
Test operation parsing
tools4origins Sep 22, 2020
8da031e
Resolve conflict in function name with pytest convention
tools4origins Sep 22, 2020
4d02d8f
Use new utils in tests
tools4origins Sep 22, 2020
dfa899a
Fix unary operation lookup
tools4origins Sep 22, 2020
f9fe965
Fix implementation of not
tools4origins Sep 22, 2020
d5ba0b1
Flush the correct streams in debug mode
tools4origins Sep 22, 2020
9d05b0c
Test unary operations
tools4origins Sep 22, 2020
2d60bae
Handle aliases
tools4origins Sep 22, 2020
97e0c69
Implement basic function call
tools4origins Oct 3, 2020
e08b7d1
Extract sql parsing in a dedicated package
tools4origins Oct 26, 2020
9245ff0
Align Expression APIs with SQL syntax
tools4origins Nov 21, 2020
1371dcb
Rename expression registering class
tools4origins Nov 21, 2020
a1d5a17
Fix instr implementation
tools4origins Nov 21, 2020
ff90059
Align code with sql parser extraction
tools4origins Nov 21, 2020
a1727e9
Fix column reference handling
tools4origins Nov 21, 2020
fb4cdf2
Rely on expression for function call
tools4origins Nov 21, 2020
0b4416d
Directly use Cast in expressions instead of wrapping column method
tools4origins Nov 21, 2020
1ee8309
Add test on fucntion evaluation
tools4origins Nov 21, 2020
8f17ad4
Remove redundant parentheses
tools4origins Nov 21, 2020
ae13310
Add tests on expression names
tools4origins Nov 21, 2020
37c5f77
Replace star import with classic import
tools4origins Nov 21, 2020
2e9d54b
Add logs
tools4origins Nov 21, 2020
043dfb4
Reduce line length
tools4origins Nov 21, 2020
d2ee9a2
Move external module import before module import
tools4origins Nov 21, 2020
426a6d0
Enhance error handling
tools4origins Jan 30, 2021
3e673d6
Fix struct parsing
tools4origins Jan 30, 2021
ccb9109
Fix typo
tools4origins Jan 30, 2021
2fb9c7e
Specify handling of more SQL grammar context
tools4origins Jan 30, 2021
6336aac
Simplify SQL alias definition parsing by using str for identifier
tools4origins Jan 31, 2021
5345821
Fix cyclic import
tools4origins Feb 27, 2021
14039af
Upgrade pythonsqlparser
tools4origins Feb 27, 2021
242982d
Rely on sql parsing for schema parsing
tools4origins Feb 27, 2021
c5f3d3f
Remove duplicated dependency
tools4origins Feb 27, 2021
add0a9c
Sort imports
tools4origins Feb 27, 2021
c36eafb
Upgrade pylint
tools4origins Feb 27, 2021
b271ac5
Fixed pylint issues.
svaningelgem Feb 28, 2021
170184f
Merge pull request #5 from svaningelgem/feat/sqlParsing
tools4origins Feb 28, 2021
9d7d3df
Add tests on parse_ddl_string
tools4origins Mar 17, 2021
9e354ba
Add tests on type parsing
tools4origins Mar 17, 2021
42327c8
Make SqlParsingError a ParseException
tools4origins Mar 17, 2021
620105f
Enhance ErrorNodeImpl handling
tools4origins Mar 17, 2021
50c1984
Fix StructType generation
tools4origins Mar 17, 2021
41100b6
Fix some conversion from sql to python
tools4origins Mar 17, 2021
bfb6b1b
Implement parse_ddl_string
tools4origins Mar 17, 2021
a168162
Replace StructType.fromDDL by a more versatile parse_ddl_string
tools4origins Mar 17, 2021
2423ccf
tmp test
tools4origins Mar 21, 2021
6e11ec6
tmp error node imp
tools4origins Mar 21, 2021
4d14116
Enhance error handling
tools4origins Aug 29, 2021
e28e6bb
Enhance type parsing
tools4origins Aug 29, 2021
1b303ec
Fix test by applying timezone offset when needed
tools4origins Aug 29, 2021
b377556
Modify import order
tools4origins Aug 29, 2021
0f1928b
Update imports
tools4origins Aug 29, 2021
a6f9702
Align with new isort behavior
tools4origins Aug 29, 2021
e9aba34
Ensure dependencies have the same behavior across machines and time
tools4origins Aug 29, 2021
50b0611
Fix import
tools4origins Aug 29, 2021
e37339d
Adapt to new pylint behavior
tools4origins Aug 29, 2021
86ab1fa
Remove Python2 specific syntax
tools4origins Aug 29, 2021
1dd3660
Enhance types support
tools4origins Aug 29, 2021
3504914
Add types to date expressions
tools4origins Aug 29, 2021
d1f6263
Add types to array expressions
tools4origins Aug 29, 2021
2ecdde7
Add types to csv expression
tools4origins Aug 29, 2021
a110689
Add types to explode expressions
tools4origins Aug 29, 2021
020b7bd
Add types to json expression
tools4origins Aug 29, 2021
55a4d78
Add types to literals
tools4origins Sep 26, 2021
9af2a6d
Add types to column operators
tools4origins Sep 26, 2021
4029a61
Add types to user defined expressions
tools4origins Sep 26, 2021
59ebc4e
Add types to string expressions
tools4origins Sep 26, 2021
a5de2f7
Add types to order expressions
tools4origins Sep 26, 2021
37b3e69
Add types to mapper expressions
tools4origins Sep 26, 2021
91b2e3d
Fix type handling in some expression
tools4origins Sep 26, 2021
78fc641
Fix typo in error message
tools4origins Sep 26, 2021
b44adf6
Fix Column handling in arrays
tools4origins Sep 26, 2021
b87e861
Add types to field expressions
tools4origins Sep 26, 2021
6d60eb2
Clarify non deterministic column initialization code
tools4origins Sep 26, 2021
507f4f7
Apply linter guidelines
tools4origins Sep 26, 2021
2025eba
Merge pull request #8 from tools4origins/feat/sqlParsing
tools4origins Oct 31, 2021
8b801ac
Merge pull request #9 from tools4origins/feat/ExpressionTypes
tools4origins Oct 31, 2021
9eb755a
fix(dataFrameShow): Remove extra new lines
tools4origins Sep 26, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pylintrc
Expand Up @@ -3,7 +3,7 @@
variable-rgx=[a-z0-9_]{1,30}$
good-names=log

disable=invalid-name,unused-argument,too-few-public-methods,no-self-use,missing-docstring,logging-format-interpolation,too-many-instance-attributes,duplicate-code,too-many-public-methods,too-many-arguments,protected-access,too-many-lines
disable=invalid-name,unused-argument,too-few-public-methods,no-self-use,missing-docstring,logging-format-interpolation,too-many-instance-attributes,duplicate-code,too-many-public-methods,too-many-arguments,protected-access,too-many-lines,unspecified-encoding,consider-using-dict-items,consider-using-with

[FORMAT]
max-line-length=119
Expand Down
26 changes: 26 additions & 0 deletions pysparkling/sql/ast/README.md
@@ -0,0 +1,26 @@
# Python Abstract Syntax Tree for Spark SQL

This folder uses ANTLR4 to convert a SQL statement in an Abstract Syntax Tree in Python.

This AST is then transformed in the corresponding pysparkling abtrasaction.

## Example


## Recreate generated files

First, download the ANTLR complete JAR from [the ANTLR site][antlr].

[antlr]:http://www.antlr.org/

Next, install the required dev ANTLR4 Python 3 runtime package:

```
pip install antlr4-python3-runtime
```

Then, run ANTLR to compile the SQL grammar and generate Python code.

```
java -Xmx500M -cp "<path to ANTLR complete JAR>:$CLASSPATH" org.antlr.v4.Tool -Dlanguage=Python3 SqlBase.g4
```
Empty file added pysparkling/sql/ast/__init__.py
Empty file.