Deleted tags still used in source code and available in Site Kit settings

[bethanylang]

Bug Description

This was identified during Bug Bash by @jamesozzie and @techanvil.

If a tag is deleted at Tag Manager level, it's still available within the Web Data Stream lookup.

This may be something to be addressed at Tag Manager level, or it may time some time before it's no longer available.

Note that SK does switch the "Exclude from Analytics" reference from the Google Tag to the G4 tag, no issues there when checking the site's source code.

Another variation:

When trying deleting the tag, the user may be directed to delete the destination (data stream) from GA.

When you delete the destination data stream, you get the following on the Analytics settings page, with the data stream not displayed in the dropdown, which is also empty.

However, when rather than deleting the individual destination data stream, you delete the property instead, you get the following on the Analytics settings page, with the property not present in the dropdown, which is displayed empty, while the datastream is still present and displayed in the dropdown. I would have thought the datastream should no longer be available at this point.

Steps to reproduce

1. Connect GA
2. Delete the Tag
3. In the SK settings, the deleted tag is selected with no error or warning that the tag doesn't exist. The GTE tag is also evident when checking the JSON code in the site source code.
4. If you select the option for SK to create a new data stream, the new data stream is selected but the exclude from Analytics option continues to select the GA4 tag and not the new GTE tag. In addition, in the JSON code while logged in, the old (non-existent tag) remains.

Screenshots

---

Do not alter or remove anything below. The following sections will be managed by moderators only.

Acceptance criteria

The Google tag sync and mismatch notification should be updated as follows:

• When there is a current Google tag ID, but the call to retrieve the container for the measurement ID does not return a container (i.e. this unhandled case), the implication is the web data stream has been deleted. Note that a container lookup will still succeed when the permission for the destination property has been removed, which may be a glitch. We should phrase the message with a degree of ambiguity i.e. "is no longer available" as opposed to "has been deleted" in case things change here.
  • A new notification should be displayed to the user, with the following message:
    • The previously selected web data stream with measurement ID {measurementID} is no longer available. Please select a new web data stream to continue collecting data with Google Analytics 4.
    • It should have a CTA, Update Analytics settings, which should navigate to the Analytics edit settings page.
    • It should have a Maybe later dismissal button, which should dismiss the banner. There's no need to persist the dismissal as the sync won't be attempted for another hour anyway.

The Analytics edit settings page should be updated as follows for GA4:

• If there is a currently selected property (propertyID) and web data stream (measurementID):
  • If the current property exists (is returned by the backend), and the call to retrieve the list of web data streams succeeds, but the current web data stream is not contained within that list, it means the web data stream has been deleted. Note that individual data streams cannot be restored.
    • An error should be displayed to the user:
      • The previously selected web data stream with measurement ID {measurementID} is no longer available. Please select a new web data stream to continue collecting data with Google Analytics 4.
  • If the call to retrieve the list of properties succeeds, but the current property is not contained within the list, it means the property has been deleted or the permission to the property has been removed. Note that the call to retrieve web data streams for the property may still succeed, possibly due to caching on the Google side.
    • An error should be displayed to the user:
      • The previously selected property with ID {propertyID} is no longer available. Please select a new property to continue collecting data with Google Analytics 4.
• The errors described above should be displayed above the GA4 dropdowns with the usual error styling, for example:
  

Implementation Brief

Show notification

Within assets/js/modules/analytics-4/datastore/properties.js:

• Add a new isWebDataStreamNotAvailable boolean flag to state, with a corresponding setIsWebDataStreamNotAvailable() action & reducer, and isWebDataStreamNotAvailable() selector.
• In *syncGoogleTagSettings(), in the case where there's no container found, set isWebDataStreamNotAvailable to true. Allow the flow to continue so googleTagLastSyncedAtMs is updated, however don't allow hasMismatchedGoogleTagID to be set.

Within new file assets/js/components/notifications/WebDataStreamNotAvailableNotification.js, create a WebDataStreamNotAvailableNotification component.

• This should be implemented using BannerNotification.
  • The title, not specced in the AC, should have similar wording to GoogleTagIDMismatchNotification: Your Analytics 4 configuration has changed
  • The description should be as per the AC.
  • The CTA should have the label as per the AC, and on click should navigate to the Analytics settings page.
  • The dismiss label should be as per the AC.
  • As dismissals are always persisted, set dismissExpires to an arbitrary low value to ensure it expires before the banner may next be shown in an hour.

Update settings UI

Within assets/js/modules/analytics/components/settings/GA4SettingsControls.js:

• Add a similar call to the one in PropertySelect to retrieve the list of properties for the current Analytics account.
• Add a similar call to the one in WebDataStreamSelect to retrieve the list of web data streams for the current GA4 property.
• If the following conditions are met:
  • The current property exists in the list of properties.
  • The list of web data streams does not contain a data stream whose measurement ID matches the current measurementID.
  • There is no error for the getWebDataStreams() selector,
• ... render an ErrorText component above the GA4 dropdowns, with the following message as per the AC:
  • The previously selected web data stream with measurement ID {measurementID} is no longer available. Please select a new web data stream to continue collecting data with Google Analytics 4.
• If the following conditions are met:
  • The current property does not exist in the list of properties.
  • There is no error for the GA4 getProperties() selector.
• ... render an ErrorText component above the GA4 dropdowns, with the following message as per the AC:
  • The previously selected property with ID {propertyID} is no longer available. Please select a new property to continue collecting data with Google Analytics 4.

Test Coverage

• Update tests for *syncGoogleTagSettings().
• Add a Storybook story for the new WebDataStreamNotAvailableNotification component.
• Add story variants for the GA4 SettingsForm to show the new error messages.
• Add VRT scenarios for the new stories.

QA Brief

• Follow the Steps to reproduce.
  • When the property is deleted:
    • The Error message defined in AC/IB re. no longer available property should appear in Analytics Settings Edit above the GA4 dropdowns.
  • When the web data stream is deleted:
    • The Error message defined in AC/IB re. no longer available data stram should appear in Analytics Settings Edit above the GA4 dropdowns
    • Wait an hour or more and check SK dashboard. The banner notification defined in AC should be displayed.
      • Clicking Primary CTA should navigate to Analytics Settings
      • The Secondary CTA should temporarily dismiss the notification. It should be displayed again in an hour.

Changelog entry

• Enhance the experience of handling properties and web data streams which are no longer available.

[marrrmarrr]

@techanvil this seems about right to me, would be great if @aaemnnosttv could run through the scenarios as well.

[marrrmarrr]

@techanvil this seems about right to me, would be great if @aaemnnosttv could run through the scenarios as well.

[aaemnnosttv]

Thanks @techanvil – a few points to clarify, but otherwise this looks good 👍

• If the call to retrieve the list of properties succeeds, but the current property is not contained within the list, it means the property has been deleted.

It could also be that the property was shared directly before and the user no longer has access to it. We can't know for sure if the property was deleted or not just because it isn't returned from the API or not. I would suggest we use similar ambiguous language in this case as well.

With the Admin API, it is possible to include deleted properties in the response using a showDeleted parameter but I'm not sure we should go that far with this issue. It's something we could potentially consider in the future though.

to continue tracking events with Google Analytics 4

"tracking events" might be confused with event tracking (which is a specific thing), but this is really about tracking anything at all with GA. Google seems to refer to this more generally as "collecting data" so I would suggest we use language which is more consistent with what the user gets by placing the tag.

An error should be displayed to the user

Let's be a bit more specific as to where the error should be shown.

These GA4 settings should be cleared

We need a bit more clarity about this part as to what it's referring to and when this should happen. Is this referring to the saved values or just the client side state? I'm assuming you mean the latter but we should make that a bit more explicit especially since one of these happens in a banner notification and the other is in the settings edit view. Should it happen in response to an action or automatically in the background?

[techanvil]

Thanks @aaemnnosttv, you've raised some good points.

It could also be that the property was shared directly before and the user no longer has access to it. We can't know for sure if the property was deleted or not just because it isn't returned from the API or not. I would suggest we use similar ambiguous language in this case as well.

With the Admin API, it is possible to include deleted properties in the response using a showDeleted parameter but I'm not sure we should go that far with this issue. It's something we could potentially consider in the future though.

Thanks for pointing this out - I'd missed the case where properties are shared individually, making the account shareable, and then removing the permission from a shared property causes it to be removed from the list. I've updated the AC to make the message more ambiguous. Regarding the showDeleted flag too - I'd agree, we probably don't need to go that far at this point, but it's something we could consider in future.

to continue tracking events with Google Analytics 4

"tracking events" might be confused with event tracking (which is a specific thing), but this is really about tracking anything at all with GA. Google seems to refer to this more generally as "collecting data" so I would suggest we use language which is more consistent with what the user gets by placing the tag.

Good spot, I've updated the messages to use "collecting data" rather than "tracking events".

An error should be displayed to the user

Let's be a bit more specific as to where the error should be shown.

Good shout - I think these errors would be most useful shown above the GA4 dropdown controls, and have updated the AC accordingly.

These GA4 settings should be cleared

We need a bit more clarity about this part as to what it's referring to and when this should happen. Is this referring to the saved values or just the client side state? I'm assuming you mean the latter but we should make that a bit more explicit especially since one of these happens in a banner notification and the other is in the settings edit view. Should it happen in response to an action or automatically in the background?

My initial intent was for the saved values to be cleared, as they are no longer valid. However, your comment has made me reconsider and in fact, I think it would probably be better not to do this, and rather simply flag the error state to the user and update the settings in response to their action. In the banner notification case, this would allow for a potential permission to be restored, and in the settings case, it seems preferable to avoid a background save seeing as the error is displayed along with the controls and the CTA, so the usual flow should be enough to update the settings to a valid set.

As a result, I've removed the details about clearing the settings from the AC.

[aaemnnosttv]

Thanks @techanvil that all sounds good to me, thanks for updating those parts. The only question I have left is whether or not we think it's a good idea to include wording about a potentially deleted property? Site Kit doesn't allow deleting or restoring properties from our end so perhaps this would be a good candidate to link the learn more to our own documentation which could elaborate on the reasons why something might not be showing up in the dropdown choices any more?

Otherwise this looks good to go 👍

[techanvil]

Thanks @techanvil that all sounds good to me, thanks for updating those parts. The only question I have left is whether or not we think it's a good idea to include wording about a potentially deleted property? Site Kit doesn't allow deleting or restoring properties from our end so perhaps this would be a good candidate to link the learn more to our own documentation which could elaborate on the reasons why something might not be showing up in the dropdown choices any more?

Otherwise this looks good to go +1

Thanks @aaemnnosttv. Again, your comment has given me pause for thought. The wording about a potentially deleted property made sense when the message was clear about the property being deleted, but as it's not clear that is the case, it doesn't read so well to include this extra detail. I've updated the message, and I'm not sure a "Learn more" link would be so appropriate any more, as it would be a bit inconsistent and might seem a bit odd to include for this one scenario when there are a lot of other edge cases which don't get the same treatment. I've removed the part about a "Learn more" link, but if you think we should still include one, I'll create a separate issue for it as we'd need to create the support content as well and I don't think we need to get that in before launch.

I also noticed the data stream error message still contained the phrase "so it has been removed from Site Kit's Analytics configuration", which is no longer accurate, so I've updated that too. I also updated it to read "is no longer available" to be consistent.

[aaemnnosttv]

LGTM, thanks @techanvil 👍

[aaemnnosttv]

LGTM, thanks @techanvil

IB ✅

[mohitwp]

QA Update ⚠️

• Tested on dev.
• Tested scenario when web data stream is deleted.
• Tested scenario when GA4 property is deleted.
• Notice showing under edit view of Analytics settings as per AC.
• Also, banner for web stream deletion is showing on main dashboard and functional as per AC.
• Notice under analytics edit settings and on main dashboard is showing as per AC.

@kuasha420 Here we are showing notice under analytics settings edit mode. Shouldn't we show this notice also when analytics settings view mode ?

[kuasha420]

@mohitwp the AC only mentioned adding the notices in edit settings. However you make a good point about adding this to view settings for more visibility of the problem.

As this covers the ac and the suggested enhancement is a nice to have, I'll suggest you to create a separate issue for that.

Cheers

[mohitwp]

QA Update ✅

Thank you, @kuasha420. Current results are as per AC, and it mentioned showing notice in edit settings mode so I will create another ticket to show notice under analytics settings view mode.

• Tested on dev.
• Tested scenario when web data stream is deleted.
• Tested scenario when GA4 property is deleted.
• Notice showing under edit view of Analytics settings as per AC.
• Also, banner for web stream deletion is showing on main dashboard and functional as per AC.
• Notice under analytics edit settings and on main dashboard is showing as per AC.