Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/untangling imports #161

Closed
wants to merge 11 commits into from
Closed

Feat/untangling imports #161

wants to merge 11 commits into from

Commits on Mar 1, 2021

  1. Cleanup of imports.

    - change of name from `stat_counter.py` to `statcounter.py` as that's the name in pyspark.
    - Moved as much as I could into 'terminals'. Meaning modules which are _not_ depending on any others except for externals to pysparkling. There's still a lot to do here!
    - Moved stuff out from pysparkling into pysparkling.sql because the SQL code just does not belong in the root!
    - The most glaring example of this is the method `toDF` which is moved to sql.session.py and is being monkey-patched. Just like pyspark is doing it. (don't re-invent the wheel :)).
    - Moved stuff to private (to pysparkling) modules (modules starting with `_`) which are not defined in pyspark. This helped to reduce the complexity a lot.
    svaningelgem committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    ac6a161 View commit details
    Browse the repository at this point in the history
  2. Cleanup of imports.

    - change of name from `stat_counter.py` to `statcounter.py` as that's the name in pyspark.
    - Moved as much as I could into 'terminals'. Meaning modules which are _not_ depending on any others except for externals to pysparkling. There's still a lot to do here!
    - Moved stuff out from pysparkling into pysparkling.sql because the SQL code just does not belong in the root!
    - The most glaring example of this is the method `toDF` which is moved to sql.session.py and is being monkey-patched. Just like pyspark is doing it. (don't re-invent the wheel :)).
    - Moved stuff to private (to pysparkling) modules (modules starting with `_`) which are not defined in pyspark. This helped to reduce the complexity a lot.
    svaningelgem committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    4ee7d8e View commit details
    Browse the repository at this point in the history
  3. - renamed schema_utils to _schema_utils (as it's an internal module).

    - Added a script to compare pyspark & pysparkling to see what still needs to be implemented.
    svaningelgem committed Mar 1, 2021
    Configuration menu
    Copy the full SHA
    60c74d9 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    b0ed5a4 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2021

  1. Configuration menu
    Copy the full SHA
    29a7d99 View commit details
    Browse the repository at this point in the history
  2. Fix test issues.

    svaningelgem committed Mar 2, 2021
    Configuration menu
    Copy the full SHA
    cef2e7c View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2021

  1. Configuration menu
    Copy the full SHA
    1597c72 View commit details
    Browse the repository at this point in the history
  2. Provide SparkContext

    svaningelgem committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    df4b505 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bb4a3ab View commit details
    Browse the repository at this point in the history
  4. Inherit from Context.

    svaningelgem committed Mar 3, 2021
    Configuration menu
    Copy the full SHA
    f3349ff View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2d7cae6 View commit details
    Browse the repository at this point in the history