TechDocs: Deleted pages continue to show up when using an external storage

[OrkoHunter]

Feature Suggestion

When a page is deleted in the markdown source, the corresponding TechDocs site should reflect that the page is not available.

As of now, when using any of the cloud storage publishers (GCP, AWS, etc.), their publish method only uploads generated files into the bucket. This is good to make sure newly created pages are there, existing pages are modified but does not ensure that removed pages get deleted in the bucket.

Possible Implementation

Maybe delete the particular TechDocs site's folder before uploading.

[Fox32]

One problem of cleaning the folder first is that TechDocs might not work during the upload. Maybe we could have some kind of versioning where the versions are uploaded to separate folders and we just change the reference to the latest one. This would also allow to travel back to older versions of a site.

[andrewthauer]

S3 has the ability to also delete files on the target which might address this issue there. Not sure if the other storage options also have this option.

[ottosichert]

I think we should choose an approach that is platform-agnostic. This keeps the possibility of integration other external storage providers with a minimal set of operations:

• upload
• delete
• link
• rename

I'm not sure if we should keep the whole history though, so I would propose following approach:

1. Build assets for my-site
2. Abort if building was unsuccessful
3. Upload assets to external storage with unique suffix, e.g. commit hash: my-site-a1b2c3
4. If my-site is a static folder, move to my-site-pending-deletion and create link from my-site
5. Point my-site link to my-site-a1b2c3
6. Perform smoke test on my-site
7. Rollback link and abort if test was unsuccessful
8. Delete old folder

What do you think?

[OrkoHunter]

@ottosichert Interesting! When you say create a link from 'my-site' folder to 'my-site-<new>', do you mean a symlink?

I am just curious if this is supported for cloud storage providers. This says GCS doesn't support it.

[iamEAP]

Yeah, echo @OrkoHunter. This is conceptually solid, but I think the problem is going to be that we'll still ultimately be limited to working on individual objects by the various cloud providers' APIs (meaning, spotty support for folder-level operations).

[andrewthauer]

I think you would likely need to delete them recreate the entire structure to get the most breadth. A rename could work as well (if supported), but would end up being a swap with possibly 3 operations. Both cases could lead to an inconsistent state if there are failures between. However that could be just as likely for any copy/sync, except maybe though additional logic errors introduced by added orchestration.

Why not support sync where possible and then fallback to something like a destroy / create?

As for storing history, I don't see the point. The docs are source controlled already, so this is just taking up more cloud storage that is possible to recreate from git history. It might be useful at some point to support versioned docs though. However that seems to serve a different use case imo.

[ottosichert]

Thanks for the input! I would argue though that it is favourable we use only one approach almost everywhere to reduce complexity.

So replacing link with a swap-rename operation we could opt for these steps:

1. Build assets for my-site
2. Abort if building was unsuccessful
3. Upload assets to external storage with custom suffix, e.g. my-site-pending-deployment
4. Rename folder my-site to my-site-pending-deletion
5. Rename folder my-site-pending-deployment to my-site
6. Perform smoke test on my-site
7. Delete my-site-pending-deletion if test was successful, otherwise delete my-site and rename my-site-pending-deletion to my-site

I actually like that approach, as we now don't need to check for backwards compatibility, it is already included in the happy case.

Also, to optimize for manual operations in case of failures, the folder naming should be clear during any steps of the above. Let me know if you have a suggestion to improve it further!

[iamEAP]

I think you're still going to run into an issue during implementation that's a result of folder not being a first-class concept in most cloud storage providers: folders are just UI magic on top of a flat list of objects.

In order to implement a rename folder operation, you'll have to implement a way to list all objects prefixed with the "folder" anyway (to then iterate over and rename each specific object).

And if you have that list, you could just as easily load and stash it up front (as existingFiles or something), upload the built files like we do today, then compare the list of uploadedFiles with existingFiles and anything that was in existingFiles but not in uploadedFiles can safely be deleted.

[ottosichert]

Manually retrieving all objects within a folder would seem error prone, so I had a look at current APIs that would indeed support moving:

• https://docs.aws.amazon.com/cli/latest/reference/s3/mv.html
• https://cloud.google.com/storage/docs/copying-renaming-moving-objects#storage-rename-object-nodejs
• https://docs.microsoft.com/en-us/cli/azure/storage/fs/directory?view=azure-cli-latest#az_storage_fs_directory_move

[ottosichert]

cc @camilaibs @iamEAP

While having a closer look at the rename/move operations, it seems they are still performed on file-by-file basis, and hence would incur a downtime on the target TechDocs site while publishing.

Hence we decided to go for a different approach:

• Retrieve a list of files of the previous build (currently existing in the bucket)
• Upload and overwrite all new files (effectively merging both versions)
• Deleting stale files that are not present in the current build

Draft PR: #6488