chore: Establish a baseline for the number of envs disabled per project (#6807) #6819
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ct (#6807) This PR adds a counter in Prometheus for counting the number of "environment disabled" events we get per project. The purpose of this is to establish a baseline for one of the "project management UI" project's key results. ## On gauges vs counters This PR uses a counter. Using a gauge would give you the total number of envs disabled, not the number of disable events. The difference is subtle, but important. For projects that were created before the new feature, the gauge might be appropriate. Because each disabled env would require at least one disabled event, we can get a floor of how many events were triggered for each project. However, for projects created after we introduce the planned change, we're not interested in the total envs anymore, because you can disable a hundred envs on creation with a single action. In this case, a gauge showing 100 disabled envs would be misleading, because it didn't take 100 events to disable them. So the interesting metric here is how many times did you specifically disable an environment in project settings, hence the counter. ## Assumptions and future plans To make this easier on ourselves, we make the follow assumption: people primarily disable envs **when creating a project**. This means that there might be a few lagging indicators granting some projects a smaller number of events than expected, but we may be able to filter those out. Further, if we had a metric for each project and its creation date, we could correlate that with the metrics to answer the question "how many envs do people disable in the first week? Two weeks? A month?". Or worded differently: after creating a project, how long does it take for people to configure environments? Similarly, if we gather that data, it will also make filtering out the number of events for projects created **after** the new changes have been released much easier. The good news: Because the project creation metric with dates is a static aggregate, it can be applied at any time, even retroactively, to see the effects.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
1 Ignored Deployment
|
![codescene-delta-analysis[bot]](https://avatars.githubusercontent.com/in/53921?s=60&v=4){kind=small}
Comment on lines
+262
to
+267
| const projectEnvironmentsDisabled = createCounter({ | ||
| name: 'project_environments_disabled', | ||
| help: 'How many "environment disabled" events we have received for each project', | ||
| labelNames: ['project_id'], | ||
| }); | ||
|
|
There was a problem hiding this comment.
❌ Getting worse: Complex Method
MetricsMonitor.startMonitoring already has high cyclomatic complexity, and now it increases in Lines of Code from 522 to 530
thomasheartman
commented
Apr 10, 2024
There was a problem hiding this comment.
All changes in this file are only because the "yarn lint" stage in CI fails without it. It's not actually related to the PR changes I want to merge.
Tymek
approved these changes
Apr 10, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a counter in Prometheus for counting the number of
"environment disabled" events we get per project. The purpose of this is
to establish a baseline for one of the "project management UI" project's
key results.
On gauges vs counters
This PR uses a counter. Using a gauge would give you the total number of
envs disabled, not the number of disable events. The difference is
subtle, but important.
For projects that were created before the new feature, the gauge might
be appropriate. Because each disabled env would require at least one
disabled event, we can get a floor of how many events were triggered for
each project.
However, for projects created after we introduce the planned change,
we're not interested in the total envs anymore, because you can disable
a hundred envs on creation with a single action. In this case, a gauge
showing 100 disabled envs would be misleading, because it didn't take
100 events to disable them.
So the interesting metric here is how many times did you specifically
disable an environment in project settings, hence the counter.
Assumptions and future plans
To make this easier on ourselves, we make the follow assumption: people
primarily disable envs when creating a project.
This means that there might be a few lagging indicators granting some
projects a smaller number of events than expected, but we may be able to
filter those out.
Further, if we had a metric for each project and its creation date, we
could correlate that with the metrics to answer the question "how many
envs do people disable in the first week? Two weeks? A month?". Or
worded differently: after creating a project, how long does it take for
people to configure environments?
Similarly, if we gather that data, it will also make filtering out the
number of events for projects created after the new changes have
been released much easier.
The good news: Because the project creation metric with dates is a
static aggregate, it can be applied at any time, even retroactively, to
see the effects.