Distribution: 3.17 Tracking issue

[emidoots]

Status

This tracking issue is being prepared.

Workload

@unassigned

• Add an option to force plaintext (no syntax highlighting) for debugging #8433 3d

@bobheadxi: 19.00d

• Script to flag GCP and AWS resources that are unexpected / wasteful #10116 4d
• monitoring for CPU and memory usage in Kubernetes #9791 2d
• observability: Add monitoring to signal whether instance is underprovisioned or not #7529 4d
• Prevent admins from saving site config if it doesn't pass validation #10883 1d
• resources-report: add whitelist functionality for certain resources #11005 0.5d
• src config commands fail #10419 0.5d 🐛
• Make conf.DeployType() differentiate between k8s and docker compose #10399 0.5d 🐛
• observability: Alerting should be easily configured through a file or site configuration #10641 4d
• monitoring: container monitoring should exclude jaeger side-car containers #11270 0.5d 🐛
• Automate creation of the Go third-party licenses CSV #9973 2d
• Setup new Grafana alerting on k8s.sgdev.org and sourcegraph.com to go to Slack #7527 1d

@davejrt

• move away from Prometheus alertmanager on Sourcegraph.com specifically #5370

@daxmc99: 4.00d

• Warn users when Sourcegraph is out of date #9708 0.5d
• Cannot start on docker for windows (prometheus crashed) #6577
• Address golangci-lint complaints about monitoring/generator.go #10915 0.5d
• jaeger should be accessible via app proxy #10491 1d
• k8s: Deploy jaeger-agent as a Daemonset instead of sidecar container #10052 1d
• observability: server/dev should always run latest grafana/prometheus #9983 1d
• add terraform configuration for sourcegraph.com's infrastructure #10455 2d
• Add monitoring for syntax highlighting vs. gitserver perf #10049 0.5d
• observability: Add ping performance stats to frontend dashboard #9867 0.5d
• observability: Improve ping logging #9866 0.5d 🐛

@ggilmore: 3.50d

• Set ephemeral storage resource requests and limits in deploy-sourcegraph #9604 1d 👩
• sourcegraph/security-issues #68 0.5d
• RFC 141: spike: Implementation Round 2: Dhall #10936 3d 🕵️
• RFC 141: spike: Implementation Round 2: Cue #10937 3d 🕵️

@slimsag

• Deploy and release: Replace Renovate with GitHub Actions #10011 0.5d
• verify "syntax highlighting sometimes doesn't work" is fixed on customer instance #9557 1d 🐛
• release: Docker Compose in-band #10486
• Alert when repository updates are failing #9782 0.5d
• RFC 128: Investigate and propose a solution for how the Slack /release command could be implemented behind the scenes #9252 3d
• patch release process requires high context switching #7148
• observability: improve/restructure admin documentation #9773 0.5d

@uwedeportivo

• Move e2e tests into regression test suite #9814 2d
• deploy-sourcegraph: include k8s secret that's mounted into gitserver #9722 1d
• support $CUSTOMER #10942 8d

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• Pull Request

[emidoots]

End of week update

This week we did the following:

• Uwe:
  • 🎉Released v3.16.0
  • 💪Working with Dave on $CUSTOMER GHE replica, great progress so far
  • 🏃Now focused on: $CUSTOMER replication
• Dave:
  • 💪Working with Uwe on $CUSTOMER GHE replica, great progress so far
  • 💬Researching on the side "switch away from site24x7 or find a way to make it more reliable Improve reliability of Sourcegraph.com site24x7 ping alerts #10742"
  • 🏃Now focused on: $CUSTOMER replication
• Robert:
  • 🎉Merged "siteAdmin: include alerts data in bug report page siteAdmin: include alerts data in bug report page #10704"
  • 💪Working on "distribution: resources report distribution: resources report #10784" and making excellent progress despite terrible GCP/AWS APIs
  • 💪Adding cadvisor (cpu/memory metrics) to kubernetes deployment add cadvisor to deployment deploy-sourcegraph#707
  • 💪Looking into https://github.com/sourcegraph/security-issues/issues/68
  • 🏃Now focused on: above
• Geoffrey:
  • 💬Giving ghe.sgdev.org guidance to Uwe/Dave
  • 💬Guiding Robert on https://github.com/sourcegraph/security-issues/issues/68
  • 🎉Merged PR to update-docker-tags so we can rollout RFC 141 prototype to k8s.sgdev.org
  • 💪Planned next steps for RFC 141: https://github.com/sourcegraph/sourcegraph/issues/10937 and https://github.com/sourcegraph/sourcegraph/issues/10936
  • 🏃Now focused on: Pulling trigger to deploy RFC 141 prototype on k8s.sgdev.org
• Dax:
  • 💪Continued work on "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708" (turned out to be much more involved than Stephen initially thought, worked with Stephen, Joe, and others to find path forward)
  • 🎉Resolved "Cannot start on docker for windows (prometheus crashed) Cannot start on docker for windows (prometheus crashed) #6577"
  • 🏃Now focused on: Finishing up "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
• Stephen:
  • 📆 We made a first draft of our planned work for 3.17 (to be finalized by EOD Wed).
  • 💬Planning $CUSTOMER deployment and replica
  • 💬Joining $CUSTOMER2 deployment call and providing scaling advice
  • 💬Interviewed ~3 candidates
  • 💪Reviewing ~25 PRs and helping with ongoing-work for Robert, Dax, Dave, Uwe, Geoffrey, Rijnard, Eric, Keegan, Quinn.
  • 🎉Released Docker Compose v3.16.0
  • 🏃Now focused on: Finalizing 3.17 planning, planning RFC 141 w/Geoffrey, giving direction to Robert & Dax

Legend:

• 💬 Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards

cc @christinaforney @nicksnyder

[emidoots]

End of week update

This week we did the following:

• Uwe & Dave:
  • 🎉Identified issues in (1) indexing (2) repository status page (3) postgres resource advice (4) indexer (fixed by Tomas, thank you!); see bigdata and rfc151 issue label
  • 🎉New ghe-feeder tool and handbook entry
  • 💪$CUSTOMER GHE and Sourcegraph replicas set up. 178k of 210k repos cloned (1k+ orgs)
  • 💪Assisting $CUSTOMER in loading repos
  • 🏃Now focused on: above
• Robert:
  • 🎉 Resource reporting bot (see #distribution slack) distribution: resources report #10784
  • 🎉 Resource reporting handbook docs
  • 🎉Improvements to resource reporting (working on more)
  • 🎉Cadvisor to Kubernetes deployments (container CPU/memory metrics) add cadvisor to deployment deploy-sourcegraph#707
  • 💪Mostly finished https://github.com/sourcegraph/security-issues/issues/68
  • 🔎Updating dot-com deployment: https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/2642
  • 🔎Adding over/under-provisioning alerts/monitoring to all deployments: https://github.com/sourcegraph/sourcegraph/pull/11082
  • 🔎Fixed src config: migrate Configuration usage to Settings src-cli#211
  • 🏃Now focused on: https://github.com/sourcegraph/sourcegraph/issues/10641 and https://github.com/sourcegraph/sourcegraph/issues/10399
• Geoffrey:
  • 🎉Set up automated builds and homebrew for update-docker-tags.
  • 🎉Added generation checks to k8s.sgdev.org CI pipeline
  • 🎉Deployed RFC 141 prototype to k8s.sgdev.org for dogfooding/comparison
  • 💪Began work on "spike: RFC 141 Implementation Round 2: Cue RFC 141: spike: Implementation Round 2: Cue #10937"
  • 🏃Now focused on: RFC 141
• Dax:
  • 🎉Exorcised golanci-lint errors Exorcise golanci-lint errors #10949
  • 🎉Made Jaeger easier for customers to access (no more port-forwarding) Jaeger reverse proxy working  #11002
  • 💬Investigated moving Jaeger to daemonset deployment model and discussed tradeoffs with team https://github.com/sourcegraph/sourcegraph/issues/10052
  • 🔎Waiting for Stephen's final review on "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
  • 🏃Now focused on: server/dev should always run latest grafana/prometheus Prevent releasing server without latest grafana/prometheus changes #9983
• Stephen (not working Mon/Tue):
  • 🎉 Finalized our planned work for 3.17.
  • 🎉 Finalized RFC 141 planning w/Geoffrey
  • 🎉 Reviewed managed instances approach
  • 💪 Investigating https://github.com/sourcegraph/customer/issues/60
  • 💬 Discussed resource reporting goals with Robert
  • 💬 Discussed over-under-provisioning monitoring approach with Robert
  • 💬 Discussed Jaeger operators with Dax/Beyang
  • 💬 Answering resource estimation questions
  • 💪 Various Interviews, debriefs, syncs, and other long conversations.
  • 💪 Working with $CUSTOMER on resource advice & HA deployment model across multiple AZs
  • 💪 Assisting Uwe/Dave with $CUSTOMER when needed
  • 🏃Now focused on: Reviewing changes by Robert & Dax (Warn users when Sourcegraph is out of date #9708); planned work (sourcegraph-intellij changes for $CUSTOMER)

Legend:

• 💬Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards
• 🔎In review

cc @christinaforney @nicksnyder

[emidoots]

https://github.com/sourcegraph/sourcegraph/issues/11122

[emidoots]

(deferred) End of week update

Last week we did the following:

• Uwe & Dave:
  • 🎉$CUSTOMER Sourcegraph replica has ~400k repos cloned/indexed.
  • 💪Assisting $CUSTOMER in loading repos, minikube issues, collecting feedback and filing issues (timeouts, global config issues, feature requests, etc.)
  • 💪 Uwe: Investigated Zoekt issues on replica with Stephen, networking issues with Dave and Dax.
  • 💪Uwe: Providing support to multiple other customers.
  • 💪Dave: Looking at stretch issues, supporting Robert with dot-com.
  • 🏃Now focused on: above
• Robert:
  • 🎉Concluded https://github.com/sourcegraph/security-issues/issues/68
  • 🎉Landed dot-com deployment update https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/2642
  • 🎉Updated documentation for updating dot-com deployments: update instructions for merging deploy-sourcegraph about#1020
  • 🎉Landed resource allow-listing https://github.com/sourcegraph/sourcegraph/pull/11107 so CI noise is removed
  • 🎉Tweaked cadvisor to prevent Prometheus OOMs in k8s
  • 🎉Landed fixes for src config: migrate Configuration usage to Settings src-cli#211
  • 💪Planned solution and started "observability: Alerting should be easily configured through a file or site configuration observability: Alerting should be easily configured through a file or site configuration #10641"
  • 🔎Sent PR "resources-report: write each report to new page resources-report: write each report to new page #11316"
  • 🔎Sent PR for unifying license generation: https://github.com/sourcegraph/sourcegraph/pull/11321
  • 💬Discussing approach to "Add over/under-provisioning alerts/monitoring to all deployments: https://github.com/sourcegraph/sourcegraph/pull/11082"
  • 🏃Now focused on: https://github.com/sourcegraph/sourcegraph/issues/10641 and https://github.com/sourcegraph/sourcegraph/issues/10399
• Geoffrey:
  • 🎉Concluded "spike: RFC 141 Implementation Round 2: Cue RFC 141: spike: Implementation Round 2: Cue #10937"
  • 🎉Concluded https://github.com/sourcegraph/security-issues/issues/68
  • 💪Began work on customer-outage issue ( https://github.com/sourcegraph/sourcegraph/issues/9604 )
  • 💪Began work on "spike: RFC 141 Implementation Round 2: Dhall reduce k8s upgrade merge conflicts: RFC 141: Dhall investigation #10936"
  • 🏃Now focused on: RFC 141
• Dax:
  • 🎉Concluded investigation into moving Jaeger to daemonset deployment model https://github.com/sourcegraph/sourcegraph/issues/10052
  • 🎉Landed final configuration changes for the jaeger reverse-proxy in k8s/docker-compose
  • 💪Started work on "add terraform configuration for sourcegraph.com's infrastructure add terraform configuration for sourcegraph.com's infrastructure  #10455"
  • 💪Looking into "observability: server/dev should always run latest grafana/prometheus Prevent releasing server without latest grafana/prometheus changes #9983" and supporting tooling https://github.com/slimsag/update-docker-tags/pull/4 (trickier than expected)
  • 🏃Now focused on: above; reviewing Stephen's changes to Warn users when Sourcegraph is out of date #9708
• Stephen (not working Mon/Tue):
  • 🎉Concluded investigation of https://github.com/sourcegraph/customer/issues/60
  • 🎉$CUSTOMER HA deployment set up successfully.
  • 🎉Merged "unify update site alerts + remove animated background from alerts unify update site alerts + remove animated background from alerts #11326"
  • 💬Many Interviews/debriefs/syncs/discussions (cloud security, $CUSTOMER deployment with Uwe/Dave/Dan, etc.) and code reviews.
  • 💪Extensive review & changes to "Warn users when Sourcegraph is out of date Warn users when Sourcegraph is out of date #9708"
  • 💪Started the tedious process of writing a license report for syntect_server: https://github.com/sourcegraph/sourcegraph/issues/11269
  • 💪Started work on broader distribution planning / prioritization for giving context to Gonzalo
  • 🏃Now focused on: Reviewing changes by Robert & Dax; sourcegraph-intellij changes for $CUSTOMER, planned work

Legend:

• 💬Discussion & advice
• 💪Progress
• 🎉Completed
• 🏃Working towards
• 🔎In review

cc @christinaforney @nicksnyder