Connectors metrics

[esloho]

Related to [ch60097] https://app.clubhouse.io/cartoteam/story/60097/instrument-connectors-with-business-metrics

This PR involves several events sent to PubSub, all related to connectors.

Acceptance
All the following actions should trigger the creation of the corresponding event in PubSub and then an entry in the corresponding table in BigQuery. The import/sync actions should be tested for all the connectors types, or at list examples from file connectors (i.e. google drive) and db connectors (i.e. postgres, mysql...).

1. Import a dataset/map through a connector in cartodb
   Event: map_created / dataset_created containing the user data and import info (connector, import_duration, data_from, data_size, sync_enabled)

2. Failed import through a connector in cartodb
   Event:import_failed containing the user data and import info (connector, import_duration, data_from, sync_enabled) and the error code

3. Failed sync process (i.e. when selecting the sync option in an import done through a connector in cartodb)
   Event: sync_failed containing the user data and the sync info (connector and error code)

4. Enable a feature flag (i.e. "carto-connectors") for all the cases:
   4.a) a particular user
   4.b) an org
   4.c) all the users
   Event: feature_flag_updated with the user data and the feature flag info (name and state: "enabled")

5. Disable a feature flag (i.e. "carto-connectors") for the cases:
   5.a) a particular user
   5.b) an org
   Event: feature_flag_updated with the user data and the feature flag info (name and state: "disabled")

[alrocar]

@jgoizueta are you doing the acceptance for this one? just saying because I can take a look if you want, just let me know.

[jgoizueta]

I've been testing this in staging and it seems to be working fine. I haven't tested all the cases (for technical limitations in staging right now and for lack of time), but I think this can be safely deployed and it will be easier to finish the tests in production.

Here is the staging testing procedure for future reference:

To test this in staging I used the cartodb-on-gcp-metrics metrics; I created a topic "test-import" and and subscription "conn-metrics-test" on it. I then generated a key for the existing "esloho-test" service account, which already had needed permissions (BigQuery Admin and Pub/Sub Admin roles).

For staging tests it's easier to use a dedicated for this, since we have only one host to configure.

Using a dedicated, I placed the service account credentials in /home/ubuntu/cartodb-on-gcp-metrics-86d83a5f9b6f.json and then I changed the configuration in /home/ubuntu/www/production.cartodb.com/current/config/app_config.yml:

  pubsub:
    project_id: "cartodb-on-gcp-metrics"
    topic: "test-import"
    credentials: "/home/ubuntu/cartodb-on-gcp-metrics-86d83a5f9b6f.json"

pubsub messages are sent from the tracker resqueue queue, which is handled by
the resque-users processes, so we need to restart it to take the new configuration:

sudo monit restart resque-users-0

Then we're ready to test and check by pulling the subscription projects/cartodb-on-gcp-metrics/subscriptions/conn-metrics-test

[alrocar]

Moving to deployment, we can do it next Monday.

Thanks! @jgoizueta ❤️

[ilbambino]

💥
thank you!!!