Skip to content
This repository was archived by the owner on Sep 30, 2024. It is now read-only.
This repository was archived by the owner on Sep 30, 2024. It is now read-only.

⭐ Q4B1 ScaleTesting #43745

Closed
Closed
⭐ Q4B1 ScaleTesting#43745
Assignees
mucles
Labels
tracking
@jhchabran

Description

@jhchabran
jhchabran
opened on Nov 1, 2022

ScaleTesting

Problem

Now we have scale testing capabilities in place and some teams have started using it, our main problem is to increase our ability to conduct scale testing.

So far, testing permissions is lacking, making it a complete blind spot for everyone to test those. It's a notoriously difficult thing to test and is vital for customers as this usually gets in the way of adopting Sourcegraph.

Another problem is that we need more test-data, as this is the defining factor in our ability to test our product at scale. This data needs to be properly organized and should be easy to recreate in the eventuality of a mistake overwriting it while conducting some tests. This test-data should be available on the necessary variety of codehosts. What is available where is based on the needs of our users.

Finally, automation and having the ability to define test scenarios is required as it will provide a canvas to each team to start accumulating test scenarios they can reuse.

Scope

The main scope of this bet is around test-data and providing automation around running the tests themselves. While it may be tempting to think about automating the provisioning of scale testing instances, the current situation for our users is that tests are conducted manually and spontaneously when needs arise. We haven't once had a case where two users had to use the scale testing at once.

Therefore, it's tactically more sound to postpone this, and to focus on providing better test-data and automation to provide that data. Once we have this, scaling the operations horizontally will be meaningful.

Boundaries

  • Single instance only.
  • Automate running tests, but not the scale testing instance itself.
  • Limited automation for provisioning the instance: populate, snapshot and restore.
  • We maintain, but do not monitor the scale testing instance, that's up to the teams using the instance (but if you see something, tell them eh).

Definition of Done

  • We have a set of test-data matching the immediate needs of each team that agreed to participate in the effort.
  • There is a defined way of creating a scenario to run a test against scale-testing. It can run in the CI pipeline.
  • We have a cost report available somewhere (stretch: automated).
  • We produce content to broadcast the use of scale testing throughout all engineering.

Payout

  • We are able to test permissions against all code hosts.
  • All participating teams are able to conduct tests on the scale testing instance, pertaining to their domain.
  • A runner for test scenarios against the scale testing instance is available, and participating teams can start writing those. At least two teams have written scenarios.
  • Everyone in engineering is aware of how to use scale-testing.

Approach

  • We double down on test-data generation, as it proved to be delicate and slow in some cases.
  • We port remaining dev-ops tasks, as they're added as we go.
  • We set strong expectations toward our users, we escalate to EM level to sort out if things are not moving fast enough.

Tracked issues

@unassigned

Completed

@Piszmog

Completed

@burmudar

Completed

@davejrt

Completed

@jhchabran

Completed

@kalanchan

Completed

@kopancek

Completed

@marekweb

Completed

@miveronese

Completed

@mucles

Completed

@sanderginn

Completed

Legend

  • 👩 Customer issue
  • 🐛 Bug
  • 🧶 Technical debt
  • 🎩 Quality of life
  • 🛠️ Roadmap
  • 🕵️ Spike
  • 🔒 Security issue
  • 🙆 Stretch goal

Metadata

Metadata

Assignees

Labels

tracking

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions