Move internal infra to production env

[benhylau]

^{This initial comment is collaborative and open to modification by all.}

Task Summary

🎟️ Re-ticketed from: #
🗣 Loomio: N/A
📅 Due date: N/A
🎯 Success criteria: Develop a process for @hyphacoop/infrastructure-wg to ensure reliability of our hosted services.

Discuss expectations and processes for:

• Spin-up
• Monitoring
• Backup
• Upgrade
• Rollback
• Incident response

This will probably involve moving pieces of our infrastructure into source control to start, and discussion of [ development vs. staging vs. production ] environments, adoption of monitoring and notification systems, and move us towards infra-as-code and automation tools that will let us spin up service instances rapidly.

+---------+   +---------+                    +-----------------------+
| Fact    |   | Service | (dev, stg, prd)    | Dedicated   | Proxmox |
| Finding +-->| Config  +------------------->| Server      |---------|
| Form    |   | Store   | REPO               |             |     KVM |
+---------+   +---------+  Ansible provision +-----------------------+
                           Service monitor

Details here.

To Do

• list of things we are running and where they are at current state, check with service inventory but also include "new stuff" like BBB
• compare and propose early hosting providers and what products sheet 🔒
  • recommend a server from soyoustart @YurkoWasHere
  • 2020-05-28 update sheet with offerings after all hands... "update spreadsheet with koumbit and actual soyoustart server" @YurkoWasHere
  • @YurkoWasHere fill row 33
  • inquire with koumbit @benhylau
  • @hyphacoop/business-planning-wg assess value criteria
• @YurkoWasHere put a server spec and let bizdev counter-propose something within budget
• draft internal proposal (edit: made Milestone) with timeline @benhylau
• deployment docs and estimate of how much time it'd take to automate (e.g. on ansible)

[ASoTNetworks]

Service inventory have been updated with the latest services.

[ASoTNetworks]

Here are some hosting providers and products we discussed:

Dedicated

So you Start (OVH):
E3-SAT-1-32 - $57.99 CAD/mo

• Canadian location available
• Intel Xeon E3-1245v2 @3.4GHz (4 cores 8 threads)
• 32GB DDR3 1333 MHz
• 3x 2TB SATA
• Unlimited Bandwidth
• 250 Mbps guaranteed
• 1 + 16 IPv4 for free $1.50/mo for additional
• Includes 100G FTP backup
• Only one IPv6 subnet /64
• Setup fees: $28.99CAD for subscription less than 6 months

TeraSwitch (Only US location):
X5650 Dedicated Server - $50.00 USD/mo

• We have contact to get direct support
• No setup fees
• 2x Intel Xeon X5650 @2.66GHz (12 cores 24 threads)
• 32GB DDR3
• 2x2 TB HDD
• 1 Gbps guaranteed
• 10TB Bandwidth
• 1 IPv4 + ($1.50 USD for each extra IP, comes in subnets only)
• IPv6 block up to /48 free
• Will have KVM access in the future

E3-1270v6 Dedicated Server - $69.00 USD/mo

• We have contact to get direct support
• No setup fees
• 1x Intel Xeon E3-1270v6 @3.80GHz (4 cores 8 threads)
• 32GB DDR4
• 2x 240 GB SSD (can be swapped with 2x 2TB HDDs without cost)
• 1 Gbps guaranteed
• 30TB Bandwidth
• 1 IPv4 + ($1.50 USD for each extra IP, comes in subnets only)
• IPv6 block up to /48 free
• Will have KVM access in the future

VPS:

DigitalOcean:

• Have Canadian location
• Plans starts at $5 USD/mo

TeraSwitch

• Only US location
• Plans starts at $4 USD/mo
• We have contact for direct support

[ASoTNetworks]

• Here is the deployment doc for Nextcloud: https://hackmd.io/Nt-veeH7Sq-onr8dApEHhA?view
  (see Try using NextCloud to coordinate project #239)
• Jitsi setup doc is in our organizing-private repo as it contains some internal configuration data.
• Doc we used for installing BBB https://docs.bigbluebutton.org/2.2/install.html (see Tuning our publicly available BBB instance #246)
• Matrix
• IPFS live streaming
• SSB Pub wip through https://github.com/ci-lg/p2p-community-infrastructure
• Others: Asterisk, VPN, Email

I still have to learn Ansible to give a time on how long it will take.

Edited: @benhylau added resources

[dcwalk]

Can we include looking at companies that provide more ethical (and sustainable) hosting practices?
Criteria could include:

• good employers
• green energy
• data sovereignty (e.g., stuff doesn't transit in the US so much) -- legal implications here

Some examples of hosts:

• https://www.ethicalhost.ca/ (not exactly a fit for some offerings)
• https://whc.ca/canadian-cloud-servers

There are more, some places aggregate lists: https://b2evolution.net/web-hosting/green/

[benhylau]

Target Architecture

Drafted this with @llunaCreixent.

Admin access to all hosts should be accessed from behind a VPN, with Service VMs exposing selective ports to the public Internet. Controller VM is the control centre for all manual/CI interactions with Service VMs.

+----------------------------------------------+
| Dedicated Server                             |
|                                              |
| +---------------+ +-------------------+----+ |
| | Controller VM | | dev | Service VMs | .. | |
| |---------------| +-------------------+----+ |
| | Repositories  | +-------------------+----+ |
| | CI builder    | | stg | Service VMs | .. | |
| +---------------+ +-------------------+----+ |
| +---------------+ +-------------------+----+ |
| | Monitoring VM | | prd | Service VMs | .. | |
| +---------------+ +-------------------+----+ |
+----------------------------------------------+

Repositories:

• PLAYbook repo (Ansible playbook)
• CONFiguration repo (client config and credentials, build-dependent on PLAY merges)

Environments & Workflow:

1. <service>-<uuid>.dev.hypha.systems is deployed anywhere from behind VPN
2. <service>-<org>.stg.hypha.systems is deployed by CI manually/on-merge from CONF/staging branch
3. Manually verify staging features on stg environment before merging to master
4. <service>.<org>.hypha.systems (or custom domain), aka. prd, is deployed by CI manually/on-merge from CONF/master branch

MVP Architecture

These are the differences from above target architecture:

• PLAY repository in public repo (this will always be open source and public), manually clone to Controller VM for now
• CONF repository in private repo, manually clone to Controller VM and deploy to stg and prd environment (CI as stretch goal)
• Minimal monitoring (not as a separate VM but prepare for Prometheus, etc.)
• No support for custom domains

Open Questions

1. What exactly goes into PLAY vs. CONF
2. How to "add Inventory" (scale beyond one dedicated server)
3. Backup strategy (MVP and beyond)
4. How to ensure others can make use of PLAY despite not having CONF (a sample?)

[benhylau]

@ASoTNetworks some scenarios to consider, would like a step-by-step from you:

1. Provision 3 new Jitsi instances
2. Update firewall rules on 2 of 3 Jitsi instances
3. Dedicated server self-destruct and we need to re-instantiate the whole stack and with backup

[benhylau]

Adding to list of providers, a Montreal based coop we met at meet.coop's call https://www.koumbit.org/en/services/vps

[benhylau]

• @benhylau add summary of infra meeting discussion to here

Internal workflow and their current repos

• label sync uses hyphacoop/organizing https://github.com/hyphacoop/organizing/tree/master/.github/workflows
• currently no-op but with some future plans in hyphacoop/worker-coop-scripts
• shortlinks
  • code from hyphacoop/shortlinks
  • csv from hyphacoop/shortlinks-site
• unused chatbot uses hyphacoop/hyphacoop-chatbot
• deprecated shortlinks hyphacoop/spreadsheet2shortlinks (archived)
• vision board uses hyphacoop/vision.hypha.coop

Proposal by @benhylau

Updated after reviews by @hyphacoop/infrastructure-wg

1. Consistent namespace for infra repos that hold generic code without configurations (e.g. hyphacoop/infra-shortlinks, hyphacoop/infra-vision)
2. Public configuration repo that has a dependency on above, and hold any custom config and deployment-specific files (e.g. BBB room config, matrix room names, shortlinks.csv)
3. Private secrets repo that holds deployment keys and passwords (e.g. API keys, credentials)

Controller VM above would pull all the repos and deploy into prod env.

CI deploys into stg env. We can spin up VMs in dev env as needed.

[benhylau]

Timeline

• Set up a dedi with proxmox that we can carve out stg (on VPN only so we save IPs) and prod VMs (June)
• VM3: website + nginx + dns + link shortener + vision 💧 (June)
• VM11: passbolt 💧 (July)
• VM1: bbb 💧💧💧💾 (July)
• VM2: jitsi 💧💧💧 (August)
• VM5: backup server 💧💾💾 (August)
• VM6: prom monitoring 💧💾 (September)
• VM7: loomio 💧 (October)
• No plans
  • VM8: email 💧💾💾💾
  • VM9: matrix + whatsapp bridge + chatbot 💧💧💾
  • VM4: nextcloud + onlyoffice 💧💧💾💾
  • VM10: android vm 💧

[benhylau]

We need something like this:

• 4 physical cores
• 32 GB RAM
• at least 8 IPv4s (5 usable) but ideally 16, with IPv6 /64 (one subnet) but ideally /56 or /48
• unmetred bandwidth
• backup space
• located in Canada
• supports Proxmox

This is with soyoustart.

A similar system with a Canadian green host will be the Dedi Pro with whc.ca but need to add an additional $55/month for 16 IPs and 100 GB of backup space, which brings the monthly cost to $218.94. Comes with CentOS.

[benhylau]

Since we have already received the quote from Koumbit, I'd like to ask we make a final decision on server vendor on Wednesday's call.

The above soyoustart server and discount is no longer available, here are some comparable ones:

• soyoustart (cpu 9127) CAD 58
• ovh (cpu 9847) USD 60

Note that all these OVH and soyoustart servers are in the Beauharnois datacentre and hydropowered (with 2 other sources as power redundancy):

• https://www.datacenterknowledge.com/archives/2013/01/17/ovhs-big-green-hydro-powered-facility-in-quebec-opens

  Beauharnois is just outside of the island of Montreal and uses renewable energy from a hydro-electric dam located just 300 meters away from the building. It was also architected to not use air conditioning, using proprietary cooling technology developed in-house. All of this results in a claimed Power Usage Effectiveness of less than 1.1.

• https://business.financialpost.com/news/cheap-hydroelectric-power-draws-data-warehouse-companies-to-quebec-b-c

[ASoTNetworks]

After looking at the above products I like the OVH rise-1 option more as it comes with newer hardware, IPMI access, and 500G of backup space that can be mounted over Samba and NFS.

[ASoTNetworks]

I have updated our internal server comparison spreadsheet highlighted with colors of the top 3 choices we looked at.

Server comparisons sheet

[benhylau]

@ASoTNetworks can you also fix the infra ranks? They seem to not be consistent what 1-5 means. Also some of the costs seem wrong with setup cost and stuff.

[ASoTNetworks]

Fixed it and only rated the ones that we can use.

[benhylau]

This is a huge task. I have made a milestone to track this so we can close this thread off.