MinIO refactor on latest operator

[sylus]

We need to onboard onto the latest MinIO operator and while we are it should improve some of our workflow.

The following are some of the benefits:

• Rename MinIO to just standard and premium and various tenants
• For security document that each MinIO instance is in its own namespace
• No longer the need to fork MinIO
• When there are problems with MinIO we can now get tracing through the launched MinIO console
• Test MinIO on the service mesh with Mutual TLS

Actions we still have to take:

• Configure Boathouse and related tooling to point to new tenants
• Find / replace in notebook examples the new paths
• Add rewrite rules via Istio so old paths convert to new paths
• Mirror the data using MC from the old to the new tenants
• Scale down old tenant replicas

[zachomedia]

@brendangadd We need to send out a notification to users that the old MinIO instances are/will be deprecated and that they should switch to the new ones. What's the process for doing this?

[zachomedia]

I propose the following for the rollout:

1. Scale down old MinIO instances
2. Update Vault configuration for old MinIO instances to point to equivalent new version (so credentials get generated)
   • remove Pachyderm from injection
3. Restart all user workloads to get updated configuration + credentials
4. Set redirect from old hostname to new hostname (for users accessing in the browser)

[in a few weeks once users have had some time to adjust]

4. Remove the "old" instances from mounting

[ca-scribner]

fyi, I think at least some people have been using minimal and standard at the same time (probably unintentionally) so there may be conflicts for a sync. Hopefully not but mentioned in case

[brendangadd]

@zachomedia: Cool. And from our discussion, there will be multiple mounted folders in the filesystem during the transition period, all of which are backed by the new Minio instances. I think your approach is good. 👍

For notification, we can send out a notification email to all of our registered users using the emails in the web portal database.

/cc @ca-scribner

[ca-scribner]

Makes sense to me!

…

On Thu, Jan 7, 2021 at 15:08 brendangadd ***@***.***> wrote: @zachomedia <https://github.com/zachomedia>: Cool. And from our discussion, there will be multiple mounted folders in the filesystem during the transition period, all of which are backed by the new Minio instances. I think your approach is good. 👍 For notification, we can send out a notification email to all of our registered users using the emails in the web portal database. /cc @ca-scribner <https://github.com/ca-scribner> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#300 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALPFPI3NUAMZSNDZ72UK3ALSYYIFFANCNFSM4T3XYHLQ> .

[ca-scribner]

Re action plan for how we decommission and migration, before we do this we should send an email a few days before migration/removal outlining any edge cases in the migration that could cause data loss (ex: if you have a file of same name in both old and new, we will take the newer file. If you think this will cause a problem, migrate the data yourself before X date, etc)

[blairdrummond]

@zachomedia I just checked the "Scale down old replicas" box. Did we add re-write rules?

[brendangadd]

Not redirecting old URLs to new tenant. @blairdrummond notified users.