Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplifications and performance improvements #8

Merged
merged 10 commits into from
Jul 27, 2021
Merged

Conversation

dcmoura
Copy link
Owner

@dcmoura dcmoura commented Jun 5, 2021

After running a profiler did some changes and simplifications.

Highlights:

  • Use of flat lists for rows (_values is now a list instead of a list of lists)
  • Input over a single iterator (instead of list of iterators)
  • Column names are replaced by direct access to the _values list in the SQL expression (instead of instantiating variables with the column names for each row) - part of the processing of the SQL expression had to move to the processor class
  • One evaluation per line (instead of one per column)

@dcmoura dcmoura added the enhancement New feature or request label Jun 5, 2021
@dcmoura dcmoura added this to the 1st release milestone Jun 5, 2021
@dcmoura dcmoura requested a review from recharte June 5, 2021 16:52
Copy link
Collaborator

@recharte recharte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job on removing the nested lists in the processor is far more intuitive this way.

spyql/processor.py Outdated Show resolved Hide resolved
spyql/spyql.py Outdated Show resolved Hide resolved
spyql/processor.py Outdated Show resolved Hide resolved
spyql/output_handler.py Show resolved Hide resolved
spyql/spyql.py Outdated Show resolved Hide resolved
spyql/processor.py Outdated Show resolved Hide resolved
spyql/processor.py Show resolved Hide resolved
spyql/processor.py Show resolved Hide resolved
spyql/processor.py Outdated Show resolved Hide resolved
spyql/spyql.py Show resolved Hide resolved
dcmoura and others added 2 commits July 19, 2021 21:23
Co-authored-by: Diogo Recharte <drecharte@veniam.com>
Co-authored-by: Diogo Recharte <drecharte@veniam.com>
@dcmoura
Copy link
Owner Author

dcmoura commented Jul 20, 2021

Yes, I also agree, thanks! Tried to push most of that logic into pre-processing.

@dcmoura dcmoura merged commit a64b567 into master Jul 27, 2021
@dcmoura dcmoura deleted the perf_improvements branch July 27, 2021 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants