[BUG] Issue with Bulk Creation Fails for Notebooks

[amnr90]

🐛 What happened?

Hello,

I'm encountering an issue with the Terraform Fabric provider when executing a pipeline in Azure DevOps or locally The problem arises during the creation of Fabric Notebooks.

I’m currently creating 25 notebooks, and I suspected this might be causing the issue, so I added a dependency to sleep for 60 seconds and only created 1 notebooks at a time.

resource` "null_resource" "wait_batch_25" {
triggers = {
always_run = timestamp()
}
depends_on = [null_resource.wait_batch_24]
provisioner "local-exec" {
command = "sleep ${local.sleep_time}"
}
}

However, the notebooks that fail aren’t always the same, and some do get created successfully with the pipeline.

The pipeline fails with the following error:

│ Error: Provider returned invalid result object after apply

│

│ After the apply operation, the provider still indicated an unknown value

│ for

│ fabric_notebook.XXXXX.definition["notebook-content.ipynb"].source_content_sha256.

│ All values must be known after apply, so this is always a bug in the

│ provider and should be reported in the provider's own repository. Terraform

│ will still save the other known object values in the state

Additional Context:

I tested the same configuration using my own account and encountered the same error.

In the development environment, where changes are applied incrementally, the process works fine.
In UAT and production, where we perform bulk creation of all items from scratch, the error occurs, not allowing us to promote the code that we have been working on.

Please advise on any known limitations or workarounds.

🔬 How to reproduce?

No response

🏗️ Code Sample / Log

No response

📷 Screenshots

No response

📈 Expected behavior

No response

🌌 Environment (Provider Version)

1.0.0 and 1.2.0

🌌 Environment (Terraform Version)

1.10.4 and 1.12.2

🌌 Environment (OS)

Linux

📎 Additional context

No response

🔰 Code of Conduct

• I agree to follow this project's Code of Conduct.

[OrBaubergMicrosoft]

ack

[OrBaubergMicrosoft]

"In the development environment, where changes are applied incrementally, the process works fine."
can you explain what do you mean by incrementally?
do you create them manually one by one?

[amnr90]

"In the development environment, where changes are applied incrementally, the process works fine." can you explain what do you mean by incrementally? do you create them manually one by one?

Hi @OrBaubergMicrosoft what I mean is that during development, I was applying the changes gradually rather than creating all the elements in one go. At any given time, I was only modifying one or 2 notebooks instead of creating them all at once.

[OrBaubergMicrosoft]

Hi @amnr90 ,
Can you please share the HCL code you wrote?
Did you create the notebook with config or definition?

[amnr90]

Hi @OrBaubergMicrosoft,

Thanks for following up on this! Here’s an example of how I’ve been creating the notebooks using Terraform:

  display_name = "${local.notebook_prefix_silver}_sp_helper_tables"
  description  = "Create the tables necessary to do the initial loads"
  workspace_id = fabric_workspace.prepare.id
  format       = "ipynb"
  definition = {
    "notebook-content.ipynb" = {
      source = "${local.notebooks_file_path}/sp_helper_tables.ipynb"
      tokens = {
        # notebook metadata configuration
        "default_lakehouse_id"           = fabric_lakehouse.silver.id
        "default_lakehouse_name"         = fabric_lakehouse.silver.display_name
        "default_lakehouse_workspace_id" = fabric_workspace.storage.id
      }

    }

  }
}

The number and type of tokens varies from notebook to notebook. For context, here’s the overall folder structure of the project:

+---environments
|       dev.yaml
|       prd.yaml
|       uat.yaml
|       
+---scripts
|   +---airflow
|   |   +---code
|   |   \---to_server
|   |       +---dev
|   |       +---prd
|   |       +---uat
|   +---custom_libraries
|   |   +---dist
|   |   +---sp_utils
|   |   \---sp_utils.egg-info
|   +---eventstream
|   +---notebooks
|   \---pipelines
\---terraform
    |   .terraform.lock.hcl
    |   airflow.tf
    |   environment.tf
    |   eventstream.tf
    |   graphql_api.tf
    |   lakehouses.tf
    |   locals.tf
    |   notebooks.tf
    |   notebooks_with_depends_on.tf
    |   notebooks_with_depends_on_prev.tf
    |   null_sleep.tf
    |   outputs.tf
    |   pipelines.tf
    |   providers.tf
    |   sql_server.tf
    |   warehouse.tf
    |   workspaces.tf
    |   workspace_access.tf
    |   
    +---.terraform
    |   |   environment
    |   |   terraform.tfstate
    |   |   
    |   +---modules
    |   |       modules.json
    |   |       
    |   \---providers
    |       \---registry.terraform.io
    |           +---hashicorp
    |           |   +---azuread
    |           |   |   +---3.3.0
    |           |   |   |   \---windows_amd64
    |           |   |   |           LICENSE.txt
    |           |   |   |           terraform-provider-azuread_v3.3.0_x5.exe
    |           |   |   |           
    |           |   |   \---3.4.0
    |           |   |       \---windows_amd64
    |           |   |               LICENSE.txt
    |           |   |               terraform-provider-azuread_v3.4.0_x5.exe
    |           |   |               
    |           |   +---azurerm
    |           |   |   +---4.26.0
    |           |   |   |   \---windows_amd64
    |           |   |   |           LICENSE.txt
    |           |   |   |           terraform-provider-azurerm_v4.26.0_x5.exe
    |           |   |   |           
    |           |   |   \---4.32.0
    |           |   |       \---windows_amd64
    |           |   |               LICENSE.txt
    |           |   |               terraform-provider-azurerm_v4.32.0_x5.exe
    |           |   |               
    |           |   +---local
    |           |   |   +---2.5.2
    |           |   |   |   \---windows_amd64
    |           |   |   |           LICENSE.txt
    |           |   |   |           terraform-provider-local_v2.5.2_x5.exe
    |           |   |   |           
    |           |   |   \---2.5.3
    |           |   |       \---windows_amd64
    |           |   |               LICENSE.txt
    |           |   |               terraform-provider-local_v2.5.3_x5.exe
    |           |   |               
    |           |   +---null
    |           |   |   \---3.2.4
    |           |   |       \---windows_amd64
    |           |   |               LICENSE.txt
    |           |   |               terraform-provider-null_v3.2.4_x5.exe
    |           |   |               
    |           |   \---template
    |           |       \---2.2.0
    |           |           \---windows_amd64
    |           |                   terraform-provider-template_v2.2.0_x4.exe
    |           |                   
    |           \---microsoft
    |               \---fabric
    |                   +---1.0.0
    |                   |   \---windows_amd64
    |                   |           LICENSE.txt
    |                   |           terraform-provider-fabric_v1.0.0.exe
    |                   |           
    |                   \---1.2.0
    |                       \---windows_amd64
    |                               LICENSE.txt
    |                               terraform-provider-fabric_v1.2.0.exe
    |                               
    +---modules
    |   +---create_dags
    |   |       main.tf
    |   |       output.tf
    |   |       variables.tf
    |   |       
    |   \---module2
    \---terraform.tfstate.d
        \---dev
                terraform.tfstate
                terraform.tfstate.backup
                

[OrBaubergMicrosoft]

maybe I am missing something,
in your code the notebook does not depend on the null resource
why is that?

[amnr90]

Hi @OrBaubergMicrosoft, the null resource was one of the approaches I tested while trying to resolve the issue. Here’s the version using null resource

resource "fabric_notebook" "sp_helper_tables_silver" {
  depends_on = [null_resource.wait_batch_2]
  display_name = "${local.notebook_prefix_silver}_sp_helper_tables"
  description  = "Create the tables necessary to do the initial loads"
  workspace_id = fabric_workspace.prepare.id
  format       = "ipynb"
  definition = {
    "notebook-content.ipynb" = {
      source = "${local.notebooks_file_path}/sp_helper_tables.ipynb"
      tokens = {
        # notebook metadata configuration
        "default_lakehouse_id"           = fabric_lakehouse.silver.id
        "default_lakehouse_name"         = fabric_lakehouse.silver.display_name
        "default_lakehouse_workspace_id" = fabric_workspace.storage.id
      }

    }

  }
}

resource "null_resource" "wait_batch_2" {
  triggers = {
    always_run = timestamp()
  }
  depends_on = [null_resource.wait_batch_1]
  provisioner "local-exec" {
    command = "sleep ${local.sleep_time}"
  }
}

Also I tested with depends on the previous notebook.

resource "fabric_notebook" "sp_helper_tables_silver" {
  depends_on = [fabric_notebook.sp_helper_tables_bronze]
  display_name = "${local.notebook_prefix_silver}_sp_helper_tables"
  description  = "Create the tables necessary to do the initial loads"
  workspace_id = fabric_workspace.prepare.id
  format       = "ipynb"
  definition = {
    "notebook-content.ipynb" = {
      source = "${local.notebooks_file_path}/sp_helper_tables.ipynb"
      tokens = {
        # notebook metadata configuration
        "default_lakehouse_id"           = fabric_lakehouse.silver.id
        "default_lakehouse_name"         = fabric_lakehouse.silver.display_name
        "default_lakehouse_workspace_id" = fabric_workspace.storage.id
      }

    }

  }
}

Unfortunately, neither option worked. To unblock myself, I commented out all references to the fabric_notebook resource and created the notebooks as templates so I could keep the references. I uploaded the notebooks manually and added them to the Terraform state. Once everything was in place, I commented out the fabric_notebook resource, and things are now working correctly.
However, this process relied on a service account from my local machine and isn’t replicable in our Azure DevOps pipeline setup.