Skip to content

Latest commit

 

History

History
51 lines (38 loc) · 2.76 KB

migrate-synapse-spark-libraries.md

File metadata and controls

51 lines (38 loc) · 2.76 KB
Raw
title description ms.reviewer ms.author author ms.topic ms.custom ms.date
Migrate Spark libraries
Learn about migrating Spark libraries from Azure Synapse Spark to Fabric, including migration prerequisites and options.
snehagunda
aimurg
murggu
how-to
fabric-cat
ignite-2023
ignite-2023-fabric
11/15/2023

Migrate Spark libraries from Azure Synapse to Fabric

Azure Synapse Spark pool offers different feed and custom library options. In Fabric, each runtime comes with a preinstalled set of libraries (built-in libraries). However, based on your scenarios and specific needs, you can also include other libraries using environment libraries and inline libraries. To move Azure Synapse Spark pool libraries to Fabric, use environment libraries.

For Spark library considerations, refer to differences between Azure Synapse Spark and Fabric.

Prerequisites

  • If you don’t have one already, create a Fabric workspace in your tenant.
  • If you don’t have one already, create an Environment in your workspace.

Option 1: Adding Spark libraries to custom environment

You can move Spark pool libraries to an environment as follows:

  1. Open Synapse Studio: Sign-in into Azure. Navigate to your Azure Synapse workspace and open the Synapse Studio.
  2. Locate Spark libraries:
    • Go to Manage area and select on Apache Spark pools.
    • Find the Apache Spark pool, select Packages and locate the Spark libraries for the pool.
  3. Get Spark libraries: Locate the requirements.txt, environment.yml, or workspace packages installed in the pool. Get the list of installed libraries on the pool.
  4. Once you have Spark libraries, add custom Spark libraries to an environment in Fabric. Within your Environment, go to Libraries and add libraries:
    • From Public Libraries, you can upload a .yml file. You can also install libraries using PyPI and Conda.
    • From Custom Libraries, you can install libraries by using .jar/.whl/.tar.gz files.
  5. Click on Save and Publish changes.

:::image type="content" source="media\migrate-synapse\migrate-spark-libraries.png" alt-text="Screenshot showing Spark libraries.":::

Learn more on adding Spark libraries to an Environment.

Note

Note that library installation may take some time.

Related content