Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat create new metric explanation #8088

Merged
merged 14 commits into from
May 8, 2024
137 changes: 137 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,143 @@ This will allow the Appwrite community to sufficiently discuss the new feature v

This is also important for the Appwrite lead developers to be able to provide technical input and potentially a different emphasis regarding the feature design and architecture. Some bigger features might need to go through our [RFC process](https://github.com/appwrite/rfc).

## Adding New Usage Metrics

These are the current metrics we collect usage stats for:

| Metric | Description |
|--------|-------------------------------------------------|
| teams | Total number of teams per project |
| users | Total number of users per project|
| executions | Total number of executions per project |
| databases | Total number of databases per project |
| collections | Total number of collections per project |
| {databaseInternalId}.collections | Total number of collections per database|
| documents | Total number of documents per project |
| {databaseInternalId}.{collectionInternalId}.documents | Total number of documents per collection |
| buckets | Total number of buckets per project |
| files | Total number of files per project |
| {bucketInternalId}.files.storage | Sum of files.storage per bucket (in bytes) |
| functions | Total number of functions per project |
| deployments | Total number of deployments per project |
| builds | Total number of builds per project |
| {resourceType}.{resourceInternalId}.deployments | Total number of deployments per function |
| executions | Total number of executions per project |
| {functionInternalId}.executions | Total number of executions per function |
| files.storage | Sum of files storage per project (in bytes) |
| deployments.storage | Sum of deployments storage per project (in bytes) |
| {resourceType}.{resourceInternalId}.deployments.storage | Sum of deployments storage per function (in bytes) |
| builds.storage | Sum of builds storage per project (in bytes) |
| builds.compute | Sum of compute duration per project (in seconds) |
| {functionInternalId}.builds.storage | Sum of builds storage per function (in bytes) |
| {functionInternalId}.builds.compute | Sum of compute duration per function (in seconds) |
| network.requests | Total number of network requests per project |
| executions.compute | Sum of compute duration per project (in seconds) |
| network.inbound | Sum of network inbound traffic per project (in bytes)|
| network.outbound | Sum of network outbound traffic per project (in bytes)|

> Note: The curly brackets in the metric name represents a template and is replaced with a value when the metric is processed.

Metrics are collected within 3 scopes Daily, monthly, an infinity. Adding new usage metric in order to aggregate usage stats is very simple, but very much dependent on where do you want to collect
statistics ,via API or via background worker. For both cases you will need to add a `const` variable in `app/init.php` under the usage metrics list using the naming convention `METRIC_<RESOURCE_NAME>` as shown below.

```php
// Usage metrics
const METRIC_FUNCTIONS = 'functions';
const METRIC_DEPLOYMENTS = 'deployments';
const METRIC_DEPLOYMENTS_STORAGE = 'deployments.storage';
```

shimonewman marked this conversation as resolved.
Show resolved Hide resolved
Next follow the appropriate steps below depending on whether you're adding the metric to the API or the worker.

**API**

In file `app/controllers/shared/api.php` On the database listener, add to an existing or create a new switch case. Add a call to the usage worker with your new metric const like so:

```php
case $document->getCollection() === 'teams':
$queueForUsage
->addMetric(METRIC_TEAMS, $value); // per project
break;
```
There are cases when you need to handle metric that has a parent entity, like buckets.
Files are linked to a parent bucket, you should verify you remove the files stats when you delete a bucket.

In that case you need also to handle children removal using addReduce() method call.

```php

case $document->getCollection() === 'buckets': //buckets
$queueForUsage
->addMetric(METRIC_BUCKETS, $value); // per project
if ($event === Database::EVENT_DOCUMENT_DELETE) {
$queueForUsage
->addReduce($document);
}
break;

```

In addition, you will also need to add some logic to the `reduce()` method of the Usage worker located in `/src/Appwrite/Platform/Workers/Usage.php`, like so:

```php
case $document->getCollection() === 'buckets':
$files = $dbForProject->getDocument('stats', md5(self::INFINITY_PERIOD . str_replace('{bucketInternalId}', $document->getInternalId(), METRIC_BUCKET_ID_FILES)));
$storage = $dbForProject->getDocument('stats', md5(self::INFINITY_PERIOD . str_replace('{bucketInternalId}', $document->getInternalId(), METRIC_BUCKET_ID_FILES_STORAGE)));

if (!empty($files['value'])) {
$metrics[] = [
'key' => METRIC_FILES,
'value' => ($files['value'] * -1),
];
}

if (!empty($storage['value'])) {
$metrics[] = [
'key' => METRIC_FILES_STORAGE,
'value' => ($storage['value'] * -1),
];
}
break;
```

**Background worker**

You need to inject the usage queue in the desired worker on the constructor method
```php
/**
* @throws Exception
*/
public function __construct()
{
$this
->desc('Functions worker')
->groups(['functions'])
->inject('message')
->inject('dbForProject')
->inject('queueForFunctions')
->inject('queueForEvents')
->inject('queueForUsage')
->inject('log')
->callback(fn (Message $message, Database $dbForProject, Func $queueForFunctions, Event $queueForEvents, Usage $queueForUsage, Log $log) => $this->action($message, $dbForProject, $queueForFunctions, $queueForEvents, $queueForUsage, $log));
}
```

and then trigger the queue with the new metric like so:

```php
$queueForUsage
->addMetric(METRIC_BUILDS, 1)
->addMetric(METRIC_BUILDS_STORAGE, $build->getAttribute('size', 0))
->addMetric(METRIC_BUILDS_COMPUTE, (int)$build->getAttribute('duration', 0) * 1000)
->addMetric(str_replace('{functionInternalId}', $function->getInternalId(), METRIC_FUNCTION_ID_BUILDS), 1)
->addMetric(str_replace('{functionInternalId}', $function->getInternalId(), METRIC_FUNCTION_ID_BUILDS_STORAGE), $build->getAttribute('size', 0))
->addMetric(str_replace('{functionInternalId}', $function->getInternalId(), METRIC_FUNCTION_ID_BUILDS_COMPUTE), (int)$build->getAttribute('duration', 0) * 1000)
->setProject($project)
->trigger();
```


## Build

To build a new version of the Appwrite server, all you need to do is run the build.sh file like this:
Expand Down
Loading