Add support for organization level metrics #12375

sarah-witt · 2022-06-16T11:46:11Z

What does this PR do?

Adds support for querying metrics from the ORGANIZATION_USAGE schema https://docs.snowflake.com/en/sql-reference/organization-usage.html

Motivation

Collecting more data

Additional Notes

Currently, although snowflake supports configuring schema, the default queries are only available in ACCOUNT_USAGE. This PR adds queries specific to ORGANIZATION_USAGE, as well as default metric groups, etc.

Some differences to note:

Almost all organization usage views have a latency of 24 hours, so the information is collected from the previous day, since the data would hardly be available if queried for the current day
The recommended organization usage collection interval is 12 hours due to this high latency, but can be configured differently
organization and account usage queries cannot be collected at the same time, they have to be in separate instances
some tables have since moved into "preview mode" after initial implementation, the column names could potentially change https://docs.snowflake.com/en/sql-reference/organization-usage/data_transfer_history.html

Review checklist (to be filled by reviewers)

Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
PR title must be written as a CHANGELOG entry (see why)
Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
PR must have changelog/ and integration/ labels attached

codecov · 2022-06-16T11:51:45Z

Codecov Report

Merging #12375 (dcabb07) into master (760fb50) will increase coverage by 0.02%.
The diff coverage is 100.00%.

Flag	Coverage Δ
snowflake	`96.46% <100.00%> (+1.54%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

apigirl

docs look good!

hithwen · 2022-06-20T10:41:24Z

snowflake/README.md

+
+By default, this integration monitors the `ACCOUNT_USAGE` schema, but can also monitor the `ORGANIZATION_USAGE` schema to monitor organization-level metrics.
+
+To collect organization metrics, you can create configure the integration by changing the schema field to `ORGANIZATION_USAGE` and increasing the `min_collection_interval` to reduce the number of queries to Snowflake, as most organization queries have a latency of up to 24 hours.


Suggested change

To collect organization metrics, you can create configure the integration by changing the schema field to `ORGANIZATION_USAGE` and increasing the `min_collection_interval` to reduce the number of queries to Snowflake, as most organization queries have a latency of up to 24 hours.

To collect organization metrics, you can configure the integration by changing the `schema` field to `ORGANIZATION_USAGE`. You should also increase the `min_collection_interval` to reduce the number of queries to Snowflake as most organization queries have a latency of up to 24 hours.

What's the recommended value for min_collection_interval?

It's 43200. I will add this here as well.

hithwen · 2022-06-20T10:43:01Z

snowflake/datadog_checks/snowflake/queries.py

@@ -52,7 +52,7 @@

 # https://docs.snowflake.com/en/sql-reference/account-usage/warehouse_metering_history.html
 WarehouseCreditUsage = {
-    'name': 'billings.warehouse.metrics',
+    'name': 'billing.warehouse.metrics',


this was wrong?

It was not incorrect, just inconsistent with other naming conventions. The name is not used anywhere besides logging. I can make this change in a separate PR if you think it's cleaner.

snowflake/README.md

hithwen · 2022-06-20T10:48:49Z

snowflake/README.md

+       - schema: ORGANIZATION_USAGE
+         min_collection_interval: 43200
+    ```
+
 #### Collecting data for multiple environments

 If you want to collect data for multiple Snowflake environments, add each environment as an instance in your `snowflake.d/conf.yaml` file. For example, if you needed to collect data for two users named `DATADOG_SYSADMIN` and `DATADOG_USER`:


and add an example at the bottom of this section to collect both account and organization metrics

Sure, I can do that!

sarah-witt · 2022-06-21T15:50:11Z

Hi @apigirl I added another section to the readme, do you mind re-reviewing?

alopezz · 2022-06-22T08:33:31Z

snowflake/README.md

+    ```yaml
+    instances:
+      - account: example-inc
+        username: DATADOG_ORG_ADMIN
+        password: '<PASSWORD>'
+        role: SYSADMIN
+        database: ORGANIZATION_USAGE
+        min_collection_interval: 3600
+
+      - account: example-inc
+        username: DATADOG_ACCOUNT_ADMIN
+        password: '<PASSWORD>'
+        role: DATADOG_ADMIN
+        database: ACCOUNT_USAGE
+        min_collection_interval: 43200
+    ```


Shouldn't it be schema rather than database here? It also looks like the min_collection_interval are reversed, I would expect the instance with ORGANIZATION_USAGE to have it set to 43200 as per the section above.

Ah yes that's right, I was copying the other example 🤦‍♀️

snowflake/assets/configuration/spec.yaml

alopezz · 2022-06-22T08:37:40Z

snowflake/datadog_checks/snowflake/check.py

            except KeyError:
                self.errors.append(mgroup)

        if self.errors:
-            self.log.warning('Invalid metric_groups found in snowflake conf.yaml: %s', (', '.join(self.errors)))
+            self.log.warning('Invalid metric_groups forfound in snowflake conf.yaml: %s', (', '.join(self.errors)))


This looks like an accidental change.

Ah yes I meant to include the schema as well

Oh wow so the test covering this was actually broken which is why I didn't catch this earlier! Good spot! https://github.com/DataDog/integrations-core/blob/master/snowflake/tests/test_unit.py#L258

alopezz · 2022-06-22T09:25:39Z

snowflake/tests/snowflake_connector_patch/_snowflake_connector_patch/tables.py

@@ -45,3 +45,79 @@
        Decimal('0.000000'),
    )
 ]
+ORGANIZATION_USAGE_IN_CURRENCY_DAILY = [('test', 'Standard', 'Compute', 'USD', Decimal('0.4'), Decimal('0.7'))]


Since we are adding these here maybe we can add assertions for these metrics on the e2e test, which is where this data is used?

Updated the e2e test!

Co-authored-by: Alex Lopez <alex.lopez.zorzano@gmail.com>

github-actions · 2022-06-22T13:21:31Z

The validations job has failed; please review the Files changed tab for possible suggestions to resolve.

…ations-core into sarah/snowflake-org-usage

buraizu

Thanks, I made a small suggestion to split up a lengthy sentence, but otherwise this looks fine from a docs perspective.

snowflake/README.md

alopezz

Almost good to go, just a little thing that concerns me for testing.

alopezz · 2022-06-23T08:40:59Z

snowflake/tests/snowflake_connector_patch/_snowflake_connector_patch/connector.py

@@ -47,7 +47,7 @@ def execute(self, query):
            # look for ACCOUNT_NAME or CONTRACT_NUMBER since some views have
            # the same name in ACCOUNT_USAGE and ORGANIZATION_USAGE
            table_prefix = ""
-            if "ACCOUNT_NAME" or "CONTRACT_NUMBER" in query:
+            if "ACCOUNT_NAME" in query or "CONTRACT_NUMBER" in query:


This looks a bit brittle. Looking a bit deeper into how the code uses the snowflake library, here's what I think it might make most sense:

Since the schema is passed when calling connect

integrations-core/snowflake/datadog_checks/snowflake/check.py

Line 185 in 7e4e927

schema=self._config.schema,

How about we pass that value in our mock down to this mock Cursor and rely on that to know whether it's an organization query or not? I think that would match the faked behavior better.

Sure, good idea. I updated to better handle multiple schemas

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

…ations-core into sarah/snowflake-org-usage

alopezz · 2022-06-23T15:43:31Z

snowflake/tests/snowflake_connector_patch/_snowflake_connector_patch/connector.py

+            # check the schema since some views have
+            # the same name in ACCOUNT_USAGE and ORGANIZATION_USAGE
+            table_prefix = ''
+            if self.schema == 'ORGANIZATION_USAGE':
+                table_prefix = 'ORGANIZATION_'


👍 Works for me! A tiny nit would be that the comment mentioning view names might be a bit confusing with the final implementation.

True, I removed the comment

* Add snowflake queries * Update and test currency usage queries * Add more queries * Complete all new queries * Clean up queries, add another test * Add more tests * Add more tests, split up test files * Add organization docs, start metadata * Remove billing prefix * Update defaults * Fix query syntax * Add more queries * Allow org usage in e2e * Clean up queries, update metadata * Update readme * fix readme * Add metric groups tests * Add currency tag * Update snowflake/README.md Co-authored-by: Julia <611228+hithwen@users.noreply.github.com> * Update readme * Update snowflake/assets/configuration/spec.yaml Co-authored-by: Alex Lopez <alex.lopez.zorzano@gmail.com> * Update readme * Sync config * Update grammar * Fix logging and tests * Add organization e2e test * Assert tags * Add credit prefix, remove count assertion * Also look for contract number as org usage * Fix syntax * Better handle multiple schemas in mock library * Update snowflake/README.md Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> * Fix style * Remove comment Co-authored-by: Julia <611228+hithwen@users.noreply.github.com> Co-authored-by: Alex Lopez <alex.lopez.zorzano@gmail.com> Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com> c9f96e3

sarah-witt added 16 commits April 26, 2022 14:21

Add snowflake queries

8950b1c

Update and test currency usage queries

70228b7

Add more queries

f11c83a

Complete all new queries

2b74656

Clean up queries, add another test

df60e3a

Add more tests

045eb50

Add more tests, split up test files

6a50a6b

Add organization docs, start metadata

ebff0e4

Remove billing prefix

d67a056

Update defaults

c91217b

Fix query syntax

0e2db1a

Add more queries

ac17fc3

Allow org usage in e2e

c80163f

Clean up queries, update metadata

8901b32

Update readme

c700573

fix readme

1f33645

ghost added documentation integration/snowflake labels Jun 16, 2022

sarah-witt added 2 commits June 16, 2022 14:58

Add metric groups tests

c2e4f0c

Add currency tag

6c92733

sarah-witt added the changelog/Added label Jun 16, 2022

sarah-witt marked this pull request as ready for review June 16, 2022 13:57

sarah-witt requested review from a team as code owners June 16, 2022 13:57

apigirl previously approved these changes Jun 16, 2022

View reviewed changes

hithwen reviewed Jun 20, 2022

View reviewed changes

snowflake/README.md Outdated Show resolved Hide resolved

hithwen reviewed Jun 20, 2022

View reviewed changes

alopezz reviewed Jun 22, 2022

View reviewed changes

Update snowflake/assets/configuration/spec.yaml

ce2b10c

Co-authored-by: Alex Lopez <alex.lopez.zorzano@gmail.com>

sarah-witt added 11 commits June 22, 2022 10:34

Update readme

2380fa8

Merge branch 'sarah/snowflake-org-usage' of github.com:DataDog/integr…

b31d167

…ations-core into sarah/snowflake-org-usage

Sync config

842d36c

Update grammar

c43f283

Fix logging and tests

38c1608

Add organization e2e test

6ad2aec

Assert tags

d2e7cfc

Add credit prefix, remove count assertion

da740de

Merge branch 'master' into sarah/snowflake-org-usage

6d56335

Also look for contract number as org usage

7e4e927

Fix syntax

b218c39

buraizu previously approved these changes Jun 22, 2022

View reviewed changes

snowflake/README.md Outdated Show resolved Hide resolved

alopezz reviewed Jun 23, 2022

View reviewed changes

Better handle multiple schemas in mock library

f1e846f

sarah-witt dismissed buraizu’s stale review via f1e846f June 23, 2022 14:37

sarah-witt and others added 3 commits June 23, 2022 10:37

Update snowflake/README.md

91d3a99

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

Fix style

14d9b64

Merge branch 'sarah/snowflake-org-usage' of github.com:DataDog/integr…

82964b3

…ations-core into sarah/snowflake-org-usage

alopezz previously approved these changes Jun 23, 2022

View reviewed changes

Remove comment

dcabb07

sarah-witt dismissed alopezz’s stale review via dcabb07 June 23, 2022 15:52

alopezz approved these changes Jun 23, 2022

View reviewed changes

sarah-witt merged commit c9f96e3 into master Jun 23, 2022

sarah-witt deleted the sarah/snowflake-org-usage branch June 23, 2022 16:07

sarah-witt mentioned this pull request Jun 28, 2022

Fix organization data transfer query #12420

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for organization level metrics #12375

Add support for organization level metrics #12375

sarah-witt commented Jun 16, 2022 •

edited

Loading

codecov bot commented Jun 16, 2022 •

edited

Loading

apigirl left a comment

hithwen Jun 20, 2022 •

edited

Loading

hithwen Jun 20, 2022

sarah-witt Jun 21, 2022

hithwen Jun 20, 2022

sarah-witt Jun 21, 2022

hithwen Jun 20, 2022

sarah-witt Jun 21, 2022

sarah-witt commented Jun 21, 2022

alopezz Jun 22, 2022

sarah-witt Jun 22, 2022

alopezz Jun 22, 2022

sarah-witt Jun 22, 2022 •

edited

Loading

sarah-witt Jun 22, 2022

alopezz Jun 22, 2022

sarah-witt Jun 22, 2022

sarah-witt Jun 22, 2022

github-actions bot commented Jun 22, 2022

buraizu left a comment

alopezz left a comment

alopezz Jun 23, 2022

sarah-witt Jun 23, 2022

alopezz Jun 23, 2022

sarah-witt Jun 23, 2022


		By default, this integration monitors the `ACCOUNT_USAGE` schema, but can also monitor the `ORGANIZATION_USAGE` schema to monitor organization-level metrics.

		To collect organization metrics, you can create configure the integration by changing the schema field to `ORGANIZATION_USAGE` and increasing the `min_collection_interval` to reduce the number of queries to Snowflake, as most organization queries have a latency of up to 24 hours.

Add support for organization level metrics #12375

Add support for organization level metrics #12375

Conversation

sarah-witt commented Jun 16, 2022 • edited Loading

What does this PR do?

Motivation

Additional Notes

Review checklist (to be filled by reviewers)

codecov bot commented Jun 16, 2022 • edited Loading

Codecov Report

apigirl left a comment

Choose a reason for hiding this comment

hithwen Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarah-witt commented Jun 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarah-witt Jun 22, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jun 22, 2022

buraizu left a comment

Choose a reason for hiding this comment

alopezz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sarah-witt commented Jun 16, 2022 •

edited

Loading

codecov bot commented Jun 16, 2022 •

edited

Loading

hithwen Jun 20, 2022 •

edited

Loading

sarah-witt Jun 22, 2022 •

edited

Loading