Potential Bug: Notebook reference in Data Pipelines fail after deployment with fabric-cicd

[PeerW]

Library Version

0.1.18

What is the problem?

Hi team,

I have started to use fabric-cicd recently and came across a very odd limitation and potential bug.

A little context
I have multiple data pipelines that I use for orchestration purposes.
The Data Pipelines usually invoke multiple pipelines that e.g. ingest data from a give source.
The invoked data pipelines are using notebooks for the actual ingestion process.

The issue
Now, I use fabric-cicd to deploy from to QA and PROD workspaces.
The deployment succeeds BUT when I try to trigger my orchestration data pipeline after the deployment, they fail with the following error in both test and production workspace for each notebook.
The odd thing is, when I simply go into the data pipeline -> select the notebook-> Settings -> reselect the workspace --> reselect the notebook name, then everything works fine (see screenshot). There must be some reference/metadata that breaks behind the scene to the notebook even though it's not shown in the UI.
Does anyone experience the same with data pipelines and notebook references? Potentially a bug?

Error Details of the notebook that is triggered by the orchestration pipeline:
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - FabricHTTPException, Error value - 401 Unauthorized for url: https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails
Headers: {'Cache-Control': 'no-store, must-revalidate, no-cache', 'Pragma': 'no-cache', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/octet-stream', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'RequestId': 'afa4a0ed-0e70-452e-9ff6-5946ced7e836', 'Access-Control-Expose-Headers': 'RequestId', 'Date': 'Mon, 26 May 2025 14:19:59 GMT'}' :

When trying to inspect the snapshot, I get the following error:
_Diagnostic ID: f8ece47d-efd1-4bbc-ba22-5372d596ed1b

Timestamp: 2025-05-26T14:30:02.020Z

Message: Error: Can't get capacity id and workspace id for the artifact with id dd36d363-a619-4d8f-9f3f-f8753581f94b
s.getMWCToken (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:246651)
at async s.fetch (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:245712)
at async s.getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:245106)
at async l._getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:562557)
at async l.getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:562169)
at async h.getJobInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:433277)
at async K.fetchSparkApp (https://content.powerapps.com/resource/powerbiwfe/extension/des/spark-monitor.be1cef949bb83f2d167b.js:6:22128)
at async K._init (https://content.powerapps.com/resource/powerbiwfe/extension/des/spark-monitor.be1cef949bb83f2d167b.js:6:27060)

JSON
{
"name": "Error",
"message": "Can't get capacity id and workspace id for the artifact with id dd36d363-a619-4d8f-9f3f-f8753581f94b",
"stack": "Error: Can't get capacity id and workspace id for the artifact with id dd36d363-a619-4d8f-9f3f-f8753581f94b\n at s.getMWCToken (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:246651)\n at async s.fetch (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:245712)\n at async s.getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:245106)\n at async l._getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:562557)\n at async l.getJobDetailInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:562169)\n at async h.getJobInfo (https://content.powerapps.com/resource/powerbiwfe/extension/des/app.7561c82e44cccd80c029.js:5:433277)\n at async K.fetchSparkApp (https://content.powerapps.com/resource/powerbiwfe/extension/des/spark-monitor.be1cef949bb83f2d167b.js:6:22128)\n at async K.init (https://content.powerapps.com/resource/powerbiwfe/extension/des/spark-monitor.be1cef949bb83f2d167b.js:6:27060)"
}

[shirasassoon]

Peer Woyczechowski (@PeerW) Thanks for raising this. We will look further into the issue.

[shirasassoon]

Hi Peer Woyczechowski (@PeerW) just want better to understand the issue you are experiencing. Can you clarify the following:

1. In your feature branch/source workspace, is the data pipeline notebook activity in the source control file pointing to a notebook in the SAME workspace? If that's the case, workspace ID would be expected to show an all zero id in source control and the item id would match the logical id of the notebook in the notebook .platform file.
2. Immediately after the data pipeline is deployed, when you open it in the Fabric workspace, without making any modification to the pipeline what do you see the workspace and notebook fields pointing to?

[PeerW]

shirasassoon
Thanks for getting back to me.

1. Correct my source controlled workspace shows an all zero id for all referenced notebooks. See below

2. When I open the pipeline in the deployed workspace without making modification I see this:

The above is pointing to the correct workspace and notebook. So, visually everything looks correct. It seems to point to the current workspace and notebook. But somehow that connection needs to be refreshed before actually working.
Can you reproduce this issue?

[shirasassoon]

Peer Woyczechowski (@PeerW) Thank you for the details. I will investigate further.

[shirasassoon]

Peer Woyczechowski (@PeerW) Is the error you're seeing persistent, or do the pipelines eventually run successfully say after waiting x number of minutes before running them post-deployment?

[PeerW]

shirasassoon The error seems to be persistent.
The pipelines first start working after "refreshing" the connection by readding the notebook manually.

[PeerW]

shirasassoon
I actually solved this last week. If someone in the future experiences the same, the below might help.
The issue was the following.
Since fabric cicd deploys with the service principal, the sp needs obviously also access to items that are referenced within the notebook. Totally valid and no bug.
Basically, what happened to me was: I was retrieving SP details etc. from a key vault from within the notebook.
Turns out my SP that deployed to QA and Prod did not have sufficient access to that key vault. Hence, when running the data pipeline it fails,
It was a tricky error message that was a little misleading but since I granted the correct access, the error is gone.