Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark] always catch IllegalStateException when accessing SparkSession #2535

Merged
merged 1 commit into from Mar 26, 2024

Conversation

pawel-big-lebowski
Copy link
Contributor

Problem

IllegalStateException is not caught.

Solution

Catch the exception when calling SparkSession.active()

Note: All schema changes require discussion. Please link the issue for context.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

@boring-cyborg boring-cyborg bot added area:documentation Improvements or additions to documentation area:integration/spark labels Mar 26, 2024
@pawel-big-lebowski pawel-big-lebowski force-pushed the spark/fix-spark-session-active-calls branch from b473f10 to be91231 Compare March 26, 2024 08:45
@pawel-big-lebowski pawel-big-lebowski marked this pull request as ready for review March 26, 2024 08:47
Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
@pawel-big-lebowski pawel-big-lebowski force-pushed the spark/fix-spark-session-active-calls branch from be91231 to dcc55fe Compare March 26, 2024 09:48
@d-m-h
Copy link
Contributor

d-m-h commented Mar 26, 2024

Looks good to me.

@d-m-h d-m-h self-requested a review March 26, 2024 09:51
@mobuchowski mobuchowski merged commit 9079c15 into main Mar 26, 2024
32 checks passed
@mobuchowski mobuchowski deleted the spark/fix-spark-session-active-calls branch March 26, 2024 14:29
mobuchowski pushed a commit that referenced this pull request Mar 29, 2024
#2535)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
blacklight pushed a commit to blacklight/OpenLineage that referenced this pull request Apr 4, 2024
OpenLineage#2535)

Signed-off-by: Pawel Leszczynski <leszczynski.pawel@gmail.com>
Signed-off-by: Fabio Manganiello <fabio@manganiello.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:documentation Improvements or additions to documentation area:integration/spark
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants