Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added one internal magic to enable retry of session creation. #716

Merged
merged 3 commits into from
Jun 8, 2021

Commits on Jun 4, 2021

  1. Added one internal magic to enable retry of session creation.

    **Description**
    Fix two issues:
    1. the self.session_started[1] is set as true too early for exceptional case.
    2. the session is added too early[2] before session is actually started.
    
    Enhanced the magic kernel by adding one internal magic to allow retry session
    creation. Currently, the session creation error is treated as fatal error which
    make it impossible[3] to retry after fixing the endpoint info.
    
    [1] https://github.com/jupyter-incubator/sparkmagic/blob/bfabbb39a0249197c2c05c8efe681710fff9151b/sparkmagic/sparkmagic/kernels/kernelmagics.py#L357
    [2] https://github.com/jupyter-incubator/sparkmagic/blob/bfabbb39a0249197c2c05c8efe681710fff9151b/sparkmagic/sparkmagic/livyclientlib/sparkcontroller.py#L90
    [3] https://github.com/jupyter-incubator/sparkmagic/blob/bfabbb39a0249197c2c05c8efe681710fff9151b/sparkmagic/sparkmagic/kernels/kernelmagics.py#L350
    
    Please see the movitivation for our use cases.
    **Motivation**
    To enable user to choose an Spark cluster and connect to it at runtime(means the
    sparkmagic configuration is not generated in advance), we need to configure the endpoint
    after kernel starting and enable retry session creation after fixing the incorrect endpoint.
    
    We discussed this with Alejandro(aggFTW@) and he suggested to leverage the following _do_not_call_*
    methods to configure the endpoint dynamically and create session after specifying configuration.
    1.    _do_not_call_delete_session
    2.    _do_not_call_change_endpoint
    3.    _do_not_call_start_session
    
    To conditionally allow the session creation retry, the changes introduced a flag
    to allow retry on previous fatal of session creation. By default, the behavior
    is kept unchanged and do not allow retry on fatal.
    
    **Testing Done**
    1. Added unit test.
    2. Also ran the code in a notebook to verify.
    
    **Backwards Compatibility Criteria (if any)**
    The enhancement is activated conditionally and no compatibility issue is expected.
    edwardps committed Jun 4, 2021
    Configuration menu
    Copy the full SHA
    1b2288b View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2021

  1. Configuration menu
    Copy the full SHA
    c58be6e View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2021

  1. Clean up session when session.start() throws exception

    **Description**
    When session start() throws exception, session delete() is invoked when
    session status is not in NOT_STARTED_SESSION_STATUS to clean up resource.
    This makes it safe to add the session to session manager only when session
    is started properly.
    
    This change is to address the edge case which could cause leaking session.
    From client side we could see timeout issue but actually the session is
    already created on Livy side.
    
    **Testing Done**
    Added unit test cases.
    Also tested in a notebook.
    edwardps committed Jun 8, 2021
    Configuration menu
    Copy the full SHA
    7bd472d View commit details
    Browse the repository at this point in the history