[Spark] Fix databricks environment

[pawel-big-lebowski]

Problem

Databricks integration test is triggered manually and has not been triggered for some time. Meanwhile:

• The test is not working anymore because the openlineage-spark jar name has changed
• DatabricksEnvironmentFacetBuilder uses compileOnly class dependency which are no longer present on databricks environment which causes ClassNotFoundError

Solution

• Fix package name that is uploaded onto databricks runtime for integration test
• Use reflection to access databricks runtime properties as it is now present within classes which are not publicly available (for example com.databricks.backend.daemon.dbutils.MountInfo)
• Verify databricks property builder within integration test.
• Separate environment per databricks runtime version. If two integration tests are run at the same time, they no longer share the same tables. Suffix is added to all the names of resources (like tables, files) which were shared before.

Follow up issues:

• [Spark][Databricks] Inconsistent naming for same dataset on input and output #2543
• [Spark][Databricks] DropTableVisitor throwing an exception #2544

Please describe your change as it relates to the problem, or bug fix, as well as any dependencies. If your change requires a schema change, please describe the schema modification(s) and whether it's a backwards-incompatible or backwards-compatible change, then select one of the following:

Note: All schema changes require discussion. Please link the issue for context.

• [] Your change modifies the core OpenLineage model
• Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

One-line summary:

Checklist

• You've signed-off your work
• Your pull request title follows our guidelines
• Your changes are accompanied by tests (if relevant)
• Your change contains a small diff and is self-contained
• You've updated any relevant documentation (if relevant)
• Your comment includes a one-liner for the changelog about the specific purpose of the change (if necessary)
• You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
• You've added a header to source files (if relevant)

---

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

[pawel-big-lebowski]

Successful CI run with databricks tests

https://app.circleci.com/pipelines/github/OpenLineage/OpenLineage/9837/workflows/b539b4c5-6e53-47e8-9615-1b5e1ba90040

[mobuchowski]

One nit, besides looks good to me.