Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DO NOT MERGE] Server side snowpark #1538

Draft
wants to merge 37 commits into
base: main
Choose a base branch
from

Commits on May 3, 2024

  1. [private branch] Maintain the AST as we build up Snowpark expressions…

    …: table(), filter() (#1468)
    
    A very basic initial attempt at serializing the AST.
    
    I'm trying to maintain a parallel codebase for phases 0 and 1 for now,
    since it would be a shame to do this work twice. Once we complete and
    ship phase 0, we'll be able to drastically simplify the phase 1 client.
    
    Unlike what I mentioned before, this implementation doesn't flush
    dependencies of eagerly evaluated expressions. Instead, any client-side
    value is appended to the pending batch. This is simpler to implement and
    will likely work well, although we may need to do some dependency
    analysis on the server to ensure we don't issue unnecessary queries.
    sfc-gh-azwiegincew committed May 3, 2024
    Configuration menu
    Copy the full SHA
    7a92099 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e13fa28 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ae3469d View commit details
    Browse the repository at this point in the history

Commits on May 8, 2024

  1. [SERVER] Refresh dev branch with recent snowpark changes (#1537)

    Updates our server branch with recent snowpark changes.
    sfc-gh-lspiegelberg committed May 8, 2024
    Configuration menu
    Copy the full SHA
    4305cf2 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Update ast_pb2.py and include it in the Snowpark wheel (#1564)

    <!---
    Please answer these questions before creating your pull request. Thanks!
    --->
    
    1. Which Jira issue is this PR addressing? Make sure that there is an
    accompanying issue to your PR.
    
       <!---
       In this section, please add a Snowflake Jira issue number.
       
    Note that if a corresponding GitHub issue exists, you should still
    include
       the Snowflake Jira issue number. For example, for GitHub issue
    #1400, you should
       add "SNOW-1335071" here.
        --->
    
       Fixes SNOW-0
    
    2. Fill out the following pre-review checklist:
    
    - [ ] I am adding a new automated test(s) to verify correctness of my
    new code
       - [ ] I am adding new logging messages
       - [ ] I am adding a new telemetry message
       - [ ] I am adding new credentials
       - [x] I am adding a new dependency
    
    3. Please describe how your code solves the related issue.
    
    Update `ast_pb2.py` (already present in the repository).
    Add the `setuptools` dependencies required for development.
    Include the module path for `ast_pb2.py` in the manifest, so that the
    file makes it into the Snowpark wheel.
    sfc-gh-oplaton committed May 14, 2024
    Configuration menu
    Copy the full SHA
    e6ce3bb View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Configuration menu
    Copy the full SHA
    7fbbc51 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    0cc4fa8 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2a9b50b View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Configuration menu
    Copy the full SHA
    2e54569 View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. [feature branch] Async session request + hacked up delay to avoid rac…

    …es until we send a response on TCM creation (#1617)
    sfc-gh-azwiegincew committed May 21, 2024
    Configuration menu
    Copy the full SHA
    1446f51 View commit details
    Browse the repository at this point in the history
  2. Revert a bad merge

    sfc-gh-azwiegincew committed May 21, 2024
    Configuration menu
    Copy the full SHA
    7657d40 View commit details
    Browse the repository at this point in the history
  3. Revert a bad merge

    sfc-gh-azwiegincew committed May 21, 2024
    Configuration menu
    Copy the full SHA
    3c144f8 View commit details
    Browse the repository at this point in the history
  4. Revert a bad merge

    sfc-gh-azwiegincew committed May 21, 2024
    Configuration menu
    Copy the full SHA
    6537943 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Configuration menu
    Copy the full SHA
    29684a4 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    b086fd2 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Configuration menu
    Copy the full SHA
    8de5d28 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7275dfa View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. [Snowpark Server] Helper script to fetch protobuf from devvm (#1700)

    Run `update-from-devvm.sh` from within `src/snowflake/snowpark/_internal/proto/` with a running local devvm to update the proto file on the thin client.
    sfc-gh-lspiegelberg committed May 30, 2024
    Configuration menu
    Copy the full SHA
    f16f1cb View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    037fc60 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. Configuration menu
    Copy the full SHA
    e4a10f9 View commit details
    Browse the repository at this point in the history

Commits on Jun 10, 2024

  1. Configuration menu
    Copy the full SHA
    d30822a View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2024

  1. Configuration menu
    Copy the full SHA
    bce97c0 View commit details
    Browse the repository at this point in the history
  2. [ThinClient] Use connector with server-side-snowpark changes in setup…

    ….py (#1766)
    
    Modifies `setup.py` to use the latest HEAD of https://github.com/snowflakedb/snowflake-connector-python/tree/server-side-snowpark which includes connector changes (most notable adding the `_dataframe_ast` field for phase 0).
    
    To update your local dev environment run
    ```
    pip uninstall snowflake-connector-python -y
    python -m pip install --no-cache -e ".[development,pandas]"
    ```
    Running the pip command should show `git clone` in the logs.
    sfc-gh-lspiegelberg committed Jun 12, 2024
    Configuration menu
    Copy the full SHA
    e20a430 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    922881b View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. Configuration menu
    Copy the full SHA
    385ac60 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2024

  1. Configuration menu
    Copy the full SHA
    ce81f68 View commit details
    Browse the repository at this point in the history
  2. [SERVER-SIDE] Add test infrastructure to track all requests have data…

    …frameAST field. (#1794)
    
    Vendors snowflake vcrpy from https://github.com/Snowflake-Labs/snowflake-vcrpy (could not get install working, therefore vendoring it) with custom Snowflake changes to track requests in vendored urllib3 within the snowflake python connector.
    
    Adds decorator `check_ast_encode_invoked` (applied with `autouse=True` to all tests) which checks that every query send contains `dataframeAst` property for phase 0, and errors out together with traceback information whenever tests need to be fixed / APIs are missing that need to be encoded within the AST.
    sfc-gh-lspiegelberg committed Jun 17, 2024
    Configuration menu
    Copy the full SHA
    9eef249 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. [SERVER-SIDE] Remove temporarily Modin tests as Modin is incompatible…

    … with Python 3.8 (#1796)
    
    Remove temporarily Modin tests as Modin is incompatible with Python 3.8.
    sfc-gh-lspiegelberg committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    8d29d02 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2024

  1. [SERVER-SIDE CI] Exclude protobuf file in flake8 precommit (#1812)

    Exclude protobuf file in precommit (flake8), as protobuf does not adhere to good coding standards and a protobuf update will consequently fail precommit.
    sfc-gh-lspiegelberg committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    718fed4 View commit details
    Browse the repository at this point in the history
  2. [SERVER-SIDE CI] In update-unparser.sh script use bazel targets inste…

    …ad of sbt (#1811)
    
    Convenience script `update-unparser.sh` uses sbt build currently. With introduction of the bazel scala targets dependencies (i.e., IR changes) switch to using them to create an updates unparser.sh as IR/protobuf updates will be reflected.
    sfc-gh-lspiegelberg committed Jun 20, 2024
    Configuration menu
    Copy the full SHA
    c92e511 View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. [SERVER-SIDE CI] Temporary fix for unparser fat jar (#1817)

    Fixes issue of bazel not producing a fat jar in `scala_binary` by calling both the bazel build and the sbt build. This is a temporary workaround till we fix in bazel the rules to emit a fat jar.
    sfc-gh-lspiegelberg committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    58eaa3c View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2024

  1. Configuration menu
    Copy the full SHA
    4cecfdc View commit details
    Browse the repository at this point in the history

Commits on Jun 28, 2024

  1. Configuration menu
    Copy the full SHA
    f513ea9 View commit details
    Browse the repository at this point in the history

Commits on Jul 1, 2024

  1. Configuration menu
    Copy the full SHA
    424bab1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6834c36 View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2024

  1. Configuration menu
    Copy the full SHA
    f7f969d View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. [SNOW-1491328] Support functions.py for AST/Snowpark IR. (#1810)

    [SNOW-1491328](https://snowflakecomputing.atlassian.net/browse/SNOW-1491328)
    
    Support AST encoding for functions (up to table functions, UDFx) in `functions.py`.
    
    Details:
    - Fix `update_from_devvm.sh` script to invoke both bazel builds required.
    - Add `check=True` to `test_ast_driver.py` to error out when subprocess fails.
    - Removes cls method `._fill_ast` from `Column` class and refactors into free method within `ast_utils.py`.
    - Introduces new `build_fn_apply` helper function to create `ApplyExpr` in IR.
    
    [SNOW-1491328]: https://snowflakecomputing.atlassian.net/browse/SNOW-1491328?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ 
    
    Co-authored-by: Arthur Zwiegincew <arthur.zwiegincew@snowflake.com>
    sfc-gh-lspiegelberg and sfc-gh-azwiegincew committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    5ce96c1 View commit details
    Browse the repository at this point in the history