Unexpected behaviour for customer defined tags after a CloudFormation stack update is completed

[gavclark]

Describe the bug

When maintaining tags for resources in AWS, several methods can be used (directly in the UI via the console, Tag Editor, CLI etc.)

In data.all , the UI presented for maintaining tags for a data.all resource is actually maintaining tags for the CloudFormation stack related to the resource (e.g., dataset)

When maintaining tags via the UI in data.all, once the CloudFormation stack update is completed, Customer defined tags (which are not in the stack tags) for the resources associated with the CloudFormation stack are removed from the resource.

There may be many use cases where resource tags are maintained outside of data.all using standard methods.

How to Reproduce

Create a resource in data.all (e.g., dataset for this example) , the same happens with pipelines. Other data.all resources have not been tested.

Post resource creation tags were added to the stack and the stack was updated.

As expected, the tags were assigned to the resources created by the stack, including the bucket:
AWS ACCOUNTS VISIBLE

Tags were then maintained outside of data.all , using the AWS Console UI for S3 (key=alt-tag-01, value=tag-s3-ui):
AWS ACCOUNTS VISIBLE

Returning the the data.all UI for the dataset, the newly applied tag is not visible (because it wasn't part of the stack):

Added another tag (key=alt-tag-03, value=another-tag-dataset-stack) in the data.all UI and updated the stack:

Post the stack update, checked the tags of the S3 bucket in the console:
AWS ACCOUNTS VISIBLE

The tag (key=alt-tag-01, value=tag-s3-ui) which was maintained in the console has now disappeared!

Expected behavior

When a stack update occurs, customer defined tags which are not part of the stack but are allocated to the resource should not be removed

Alternatively, a warning message that some customer defined tags are not in the stack and will be lost if the stack updates.

Your project

n/a

Screenshots

No response

OS

n/a

Python version

3.7

AWS data.all version

1.0

Additional context

No response

[dlpzx]

I am taking a look. Thanks for flagging this issue

[gavclark]

@dlpzx that's great thanks ! I have a question - where does data.all store the tag information for the stack ? (is it in the metadata database, and if so, which tables) . The reason for the question is that we're migrating from data hub to data all, and as I understand it, data hub didn't store the tags ? . So we'll need to figure out a way to ingest the existing tags in data hub into data.all for the relevant resources

[dlpzx]

Hi @gavclark , yes data.all stores the key-value tags in a table called 'keyvaluetag' where the targetUri is the unique identifier of the object (environmentUri, datasetUri, pipelineUri) and targetType represents the type of object ( 'environment', 'dataset'...)

class KeyValueTag(Base):
    __tablename__ = 'keyvaluetag'
    tagUri = Column(String, primary_key=True, default=utils.uuid('keyvaluetag'))
    targetUri = Column(String, nullable=False)
    targetType = Column(String, nullable=False)
    key = Column(String, nullable=False)
    value = Column(String, nullable=False)

[dlpzx]

I am changing the issue from bug to enhancement, because the implementation works as expected. From data.all point of view, changes in the data.all object in the UI trigger changes in the corresponding CDK --> CFN stack so everything is deployed and updated in Infrastructure as Code. You are bypassing this by directly creating tags on specific resources. To understand the need of this feature better, can you explain why you need tags on specific resources?

[dlpzx]

IMPORTANT: @gavclark I noticed that you uploaded images with AWS Account IDs, I removed the images from the Issue

[gavclark]

@dlpzx Thanks for removing the the AWS Account ID's , I was trying to be careful too !

In response to your other question about why have tags on resources (which are not maintained in data.all) , we have a process which tags s3 buckets which may include PII data or commercially sensitive data.

Whilst I appreciate that these could be maintained in data.all, at the moment that process is outside of data hub/data.all

[dlpzx]

I see your point. From an implementation standpoint, every time there is an update of the stack from data.all we would need to:

1. check the existing tags for each AWS resource and
2. add them to their CDK definition in the CDK-CloudFormation stack

Data.all dataset stack tags all the resources of the dataset. Would you be fine with adding the S3-specific manual tag to the rest of the resources deployed by the stack? (Like if we would add the tag from the data.all UI)
If yes, the implementation is more straight-forward because instead of changing the tag of one specific resource, we add the tag as any other data.all key-value tag.

Let me know and maybe we can design the implementation together