Skip to content

Commit

Permalink
Update adding_ping_data.md (#929)
Browse files Browse the repository at this point in the history
New link to pubsub message
  • Loading branch information
Eric Brody-Moore committed May 19, 2020
1 parent f08b6d9 commit 2d857bd
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions handbook/engineering/adding_ping_data.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,12 +28,12 @@ Treat adding new data to pings as having a very high bar. Would you be willing t
- Pings documentation: https://docs.sourcegraph.com/admin/pings
- The Site-admin > Pings page, e.g.: https://sourcegraph.com/site-admin/pings
3. Determine if any transformations/ETL jobs are required, and if so, add them to the [script](https://github.com/sourcegraph/analytics/blob/master/BigQuery%20Schemas/transform.js). The script is primarily for edge cases. Primarily, as long as zeroes or nulls are being sent back instead of `""` in the case where the data point is empty.
4. Open a PR to change [the schema](https://github.com/sourcegraph/analytics/tree/master/BigQuery%20Schemas) with Business Operations (EricB and Dan) as approvers. Keep in mind:
4. Open a PR to change [the schema](https://github.com/sourcegraph/analytics/tree/master/BigQuery%20Schemas) with Business Operations (EricBM and Dan) as approvers. Keep in mind:
- Check the data types sent in the JSON match up with the BigQuery schema (e.g. a JSON '1' will not match up with a BigQuery integer).
- Every field in the BigQuery schema should not be non-nullable (i.e. `"mode": "NULLABLE"` and `"mode": "REPEATED"` are acceptable). There will be instances on the older Sourcegraph versions that will not be sending new data fields, and this will cause pings to fail.
5. Once the schema change PR is merged, test the new schema. Contact @EricBM for this part.
- Delete the [test table](https://bigquery.cloud.google.com/table/telligentsourcegraph:sourcegraph_analytics.update_checks_test?pli=1) (`$DATASET.$TABLE_test`), create a new table with the same name (`update_checks_test`), and then upload the schema with the newest version (see "Changing the BigQuery schema" for commands). This is done to wipe the data in the table and any legacy configurations that could trigger a false positive test, but keep the connection with Pub/Sub.
- [Publish a message](https://console.cloud.google.com/cloudpubsub/topic/detail/server-update-checks-test?project=telligentsourcegraph) to Pub/Sub, which will go through [Dataflow](https://console.cloud.google.com/dataflow/jobs/us-central1/2020-02-28_09_44_54-15810172927534693373?project=telligentsourcegraph&organizationId=1006954638239) to the BigQuery test table. The message can use [this example](https://github.com/sourcegraph/analytics/blob/master/BigQuery%20Schemas/pubsub_message) as a baseline, and add sample data for the new ping data points.
- Update and publish [a message](https://github.com/sourcegraph/analytics/blob/bfe437c92456f5ddb3c0e765e14133e1e1604bfb/BigQuery%20Schemas/pubsub_message.json) to [Pub/Sub](https://console.cloud.google.com/cloudpubsub/topic/detail/server-update-checks-test?project=telligentsourcegraph), which will go through [Dataflow](https://console.cloud.google.com/dataflow/jobs/us-central1/2020-02-28_09_44_54-15810172927534693373?project=telligentsourcegraph&organizationId=1006954638239) to the BigQuery test table. The message can use [this example](https://github.com/sourcegraph/analytics/blob/master/BigQuery%20Schemas/pubsub_message) as a baseline, and add sample data for the new ping data points.
- To see if it worked, go to the [`update_checks_test`](https://bigquery.cloud.google.com/table/telligentsourcegraph:sourcegraph_analytics.update_checks_test?pli=1) table, and run a query against it checking for the new data points. Messages that fail to publish are added to the [error records table](https://bigquery.cloud.google.com/table/telligentsourcegraph:sourcegraph_analytics.update_checks_test_error_records?pli=1).

6. Once the test is successful, merge your PR from step 2.
Expand Down

0 comments on commit 2d857bd

Please sign in to comment.