SPFx Extension app package periodically fails to load JavaScript files

[sionjlewis]

Category

• [] Question
• Bug
• [] Enhancement

Expected or Desired Behavior

SPFx Extension to load when page is requested.
Steps:

• Upload SPFx Extension to the App Catalog and select Tenant wide deployment
• Run the Add-PnPCustomAction PowerShell CommandLet (for a specific site)
• Load a modern site page ALL OK

Observed Behavior

The SPFx Extension will load OK for weeks and then "out of the blue" if fails with errors like:

GET https://[tenent].sharepoint.com/sites/apps/ClientSideAssets/[guid]/extention-name-application-customizer_[guid].js net: ERR_ABORTED 404
HEAD https://[tenent].sharepoint.com/sites/apps/ClientSideAssets/[guid]/extention-name-sp-page-assembly_[guid].js 404

Steps to Reproduce

Not sure how we can reproduce this problem?

• The problem:
• • Occur generally on or just after a Tuesday (New Zealand time)
• • Occurs for multiple tenants, but generally not all at the same time
• • Affect all SPFx Extensions (when it happens)
• • Does not affect SPFx Webparts
• The JavaScript files appears to be from an older versions of the App Package (we have started a log to confirm this assumption)
• When an error occurs, the SPFx Extension does not load
• Opening the Browser Console locate the error:
• • Copy and past the JavaScript's URL into a browser window will also show a 404 error
• • Remove the File name & extension from the URL will open the container folder (no errors)
• This may not be an SPFx Extension bug but rather an Office 365 update issue?

Possible solution:

I assume the GUIDs used within the file names as for cache-busting purposes. Why not remove the GUID from file name and apply it to the requesting code as a query string parameter?

Ideas and suggestions are welcome :)

#sp-dev-fx-extensions

[msft-github-bot]

Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.

[nateforsyth]

Can confirm have seen this multiple times.

[patmill]

If you browse the directory when you get a 404, can you find the file? If you can, can you select it from the library? If so, does it load, and if so, does the URL match the original request URL?

Also - can you look at the response headers and response body of the original 404? Does anything look out of whack?

[sionjlewis]

If you browse the directory when you get a 404, can you find the file?
Yes there are files, however their names end with different GUIDs
~/ClientSideAssets/[EXT-GUID]/extension-name-application-customizer_[FILE-GUID].js

If you can, can you select it from the library?
Yes, the files within the library load without errors (OK)

If so, does it load, and if so, does the URL match the original request URL?
A. Yes the file Loads.
B. No, the files has a different GUID appended to the file name
~/ClientSideAssets/[EXT-GUID]/extension-name-application-customizer_[FILE-GUID].js

Also - can you look at the response headers and response body of the original 404?
I will have to wait for the next incident and get back to you.

Does anything look out of whack?
TBC: assuming not as the request is for a different file name.

[patmill]

That's super weird. Are you updating your solution in any way? Are you uploading code separate from the solution?

The path should be in the solution manifest, so it's really strange that it should not match. Let's try this:
1 - Go to the component manifests list in your app catalog. It can be reached at yourtenant.sharepoint.com/sites/yourappcatalog/Lists/ComponentManifests/AllItems.aspx
2 - Find the manifest of the component that is misbehaving
3 - Look at the references in that file. Do they match with the file names that are in the code assets library?
4 - if not, can you see what the last modified time is on the original solution, the entry in the manifest list, and the entry in the assets library?

Those should all be in sync for the most part, and (unless there is some strange caching at play) the only reason the code filename should change is because the solution was updated, it had a manifest change, and a new bit of code.

[nateforsyth]

Following on from what @sionjlewis has said above, it's worth noting a few other things; every time this occurs:

• there has been an update to our extensions project 1-2 days before.
• the updates are done within the SPFx project.
• the project is built and bundled using the documented gulp based processes.
• the package-solution.json version is incremented.

These next points may be a red herring, however the timing seems consistent.

• this always seems to occur between the Tuesday - Wednesday window.
• this feels like the server is performing a post-update rollback to a predefined state.
• the filename it's attempting to load appears to be old (unconfirmed).
• the filename being loaded is different to what is within the ClientSideAssets folder for the extension (as previously mentioned).

Hope this helps.

[sionjlewis]

Are you updating your solution in any way?
From our recollection, this only seems to happen on environments where the package has been updated.

Are you uploading code separate from the solution?
No, all the code is deployed using an App Package

The path should be in the solution manifest, so it's really strange that it should not match.
Agree; it's like the "system" has be partially rolled back (back-up & restore)...

Let's try this:

1 - Go to the component manifests list in your app catalog. It can be reached at yourtenant.sharepoint.com/sites/yourappcatalog/Lists/ComponentManifests/AllItems.aspx
2 - Find the manifest of the component that is misbehaving
3 - Look at the references in that file. Do they match with the file names that are in the code assets library?
4 - if not, can you see what the last modified time is on the original solution, the entry in the manifest list, and the entry in the assets library?

Thanks for the URL. Unfortunately I'm unable to reproduce the error right now, as we had to apply the a fix ASAP. If/when the error occurs in our Test environment then we will be able to follow your steps above and get back to you.

Currently our fix is to replace the existing package (i.e. re-upload it). It does not require a rebuild (or changing package version).

Those should all be in sync for the most part, and (unless there is some strange caching at play) the only reason the code filename should change is because the solution was updated, it had a manifest change, and a new bit of code.

Make sense and logically agree :)

I don't want to muddy the water; however, I have wondered if this issue is related to the SharePoint Online Update process?

Remembering back to SharePoint 2007 & 2010 On-Premises days, we use to back up a site collection's database and file system separately. If these back-ups were slightly out of sync then we would see similar issues on a restore...

In the meantime here's a screen shot:

[patmill]

I don't think SPO updates are the cause here. The GUID in that file name is generated during the build process of the solution, and a reference is added to the manifest during gulp bundle and gulp package-solution --ship . Any change to the path would come from an upload to the library, either because of an update to the solution (which cracks the solution file and copies the files) or because someone updated the files directly. My only guess is that there is a cache issue, where the solution is updated, the manifest is changed, the new files are uploaded, but the manifest is cached somewhere and an old reference is getting used.

[sionjlewis]

The GUID in that file name is generated during the build process of the solution, and a reference is added to the manifest during gulp bundle and gulp package-solution --ship .

Yes, 100% agree that is what the SPFx build process does.

Any change to the path would come from an upload to the library, either because of an update to the solution (which cracks the solution file and copies the files) or because someone updated the files directly.

Yes, also agree.
NOTE: we are not uploading the files directly :)

My only guess is that there is a cache issue, where the solution is updated, the manifest is changed, the new files are uploaded, but the manifest is cached somewhere and an old reference is getting used.

A "Cache issue" would seem a plausible explanation. However the error does not occur immediately after and update. If it did then replicating the error would be much easier...

• We have previously tried deleting the App Package and JavaScript folder (located in ClientSideAssets) before re-uploading the Package to the App Catalog. This has fixed the immediate problem, but in some cases the issue has reoccurred some time later (days or weeks).

• What we haven't checked is that the "~/Lists/ComponentManifests" items have been deletes too.

If it's a "Cache issue", any ideas on how can we fix it?

[semopz]

I've had issues with extensions before the latest SPO updates and the following steps helped (may not be the case with OP's issue).

1. As mentioned, always increment the version number of your solution before deployment.
2. Don't let SharePoint update the extension for you. Use Uninstall-PnPApp to remove the old version and then use Install-PnPApp to install the new version. I found that doing these steps through the GUI would cause issues as sometimes, the uninstall stage would still be in progress even though the GUI would indicate otherwise. The PnP Install would fail if other operations are still in progress (might be worth just waiting 30 mins after an uninstall just to be safe).
3. Whenever you install an extension "App" you also get a CustomAction added to your SPWeb. Keep track of those and make sure that when you update an extension, 1 CustomAction is deleted after uninstall and that an install adds 1 CustomAction. You can keep an eye on those with Get-PnPCustomAction

[sionjlewis]

Hi @semopz,
Great to hear your experience with this issue.

1. As mentioned, always increment the version number of your solution before deployment.

Yes / thumbs-up

2. Don't let SharePoint update the extension for you. Use Uninstall-PnPApp to remove the old version and then use Install-PnPApp to install the new version. I found that doing these steps through the GUI would cause issues as sometimes, the uninstall stage would still be in progress even though the GUI would indicate otherwise. The PnP Install would fail if other operations are still in progress (might be worth just waiting 30 mins after an uninstall just to be safe).

Interesting; we have previously had issues with the Install-PnPApp cmdlet timing out, so started using a manual process (the UI) instead. Will try PowerShell next time, to see if that resolves the issue.

I have a concern regarding "... waiting 30 mins after an uninstall ...", as this often not possible on 'live' systems. So some more thought will be required...

3. Whenever you install an extension "App" you also get a CustomAction added to your SPWeb. Keep track of those and make sure that when you update an extension, 1 CustomAction is deleted after uninstall and that an install adds 1 CustomAction. You can keep an eye on those with Get-PnPCustomAction

Excellent point, we will include that in our monitoring of the issue/bug

Thank you

Updated to checklist:

• https://[tenent].sharepoint.com/sites/[app-catalog]/Lists/ComponentManifests/
• https://[tenent].sharepoint.com/sites/[app-catalog]/ClientSideAssets/[SPFx-Ext-GUID]/
• Get-PnPCustomAction

[andrewconnell]

@sionjlewis - Seems based on your last reply this issue can be closed as resolved... correct?