Skip to content

04. Canopy Project Management

Jump to bottom
Janemesi edited this page Oct 22, 2025 · 3 revisions

Project Scoping Guidelines

The Analytics team has developed these resources to help understand and map client requirements for a project. The list is not extensive, and each document is meant as a guideline more than a hard template to follow. Feel free to take these, reference them in your project, and evolve them based on your needs:

Granting Clients Database Access

When a client needs access to a database within the Canopy shared host:

  1. Twingate permissions:

    The analyst/engineer should create Twingate permissions for the client to access the network. For more information on Twingate, refer to this documentation. The client also needs to download Twingate on their end.

  2. User creation:

    The analyst/engineer should create a user who inherits the same privileges as the database owner. Refer to this [sample case] (https://github.com/onaio/data-solutions/issues/457) for guidance. The created user credentials should be shared with the client using a password manager tool such as Bitwarden Send.

  3. DBT cloud setup:

    If the client is using DBT Cloud, they need to set up the DBT project and use the shared user credentials to connect to the database when creating their development and deployment environment.

    If the client encounters a permission error when running a job on DBT, the analyst should confirm the owner of the tables within the production schema.

    If the owner is not the same as the client user, the schema needs to be dropped so that the client can run the job successfully on DBT Cloud.

Project Closure Guidelines

Types of Project Closure

1. Scale Down

This refers to when a project is taking a pause and is scheduled to resume later. Resumption date is usually defined.

2. Temporary Shutdown

A project has come to an end and might or might not resume but the resumption date is not defined.

3. Permanent Shutdown

Project has come to an end and it will not restart. The Data Engineering and Analytics Team is responsible for the shutdown of the ETLs, databases and canopy services.

Please use the following checklist template for shutting down a project on the Canopy repo.

Nifi connectors

  • Check for and pause NiFi connectors
  • Check for connectors on Notion
  • Update Notion after you pause/disable the connectors

Beam connectors (newer projects)

  • Check beam kiwi space and stop or delete the connectors as necessary
  • Copy the connector code for each connector and include it in the issue
  • Update Notion after you pause/disable the connectors

Airbyte connectors

  • Check for and pause all project specific connectors
  • Revoke client access

PowerBI

  • Check for and pause specific PowerBI/NiFi connector

Canopy Dashboards (Superset)

  • Revoke access to discover.ona.io/superset.ona.io for clients on shared instance
  • Stop running/freeze client instance of superset - TBD need to get details

Akuko organistion/spaces

  • Revoke client access to organisation and spaces
  • Once possible, freeze the project organisation

Gisida instance

  • Stop running/freeze client instance of gisida

Canopy Database

  • Create a database backup the DB (pgdump)
  • Save and share the data dump in location determined with the client (e.g. S3)
  • Follow up with SRE on Security protocol for sharing data dump N.B.: Data dump is stored for 6 months max before deletion

Deleting old files from AWS S3

To delete old files from an entire S3 bucket;

  • Open Amazon S3 and select one bucket from the list, on which we want to enable automatic deletion of files after a specified period of time.
  • Go to Management and click on Create lifecycle rule.
  • Provide a name for the rule and select the option saying that the changes are to apply to all objects and select the checkbox that appears.
  • Select Expire current versions of objects option and enter the number of days after which the files will be deleted. Its recommended that one sets 6 months for permanent closure.
  • Click on Create rule and the automatic file deletion rule will be applied.

To delete files from one folder from S3 bucket;

  • Open Amazon S3 and select one bucket from the list, on which we want to enable automatic deletion of files after a specified period of time.
  • Go to Management and click on Create lifecycle rule.
  • Provide a name for the rule and select the option saying that the changes are to apply to all objects and select the checkbox that appears.
  • Select Limit the scope of this rule using one or more filters option and specify the name of the folder under the prefix section. For example testing/
  • Select Expire current versions of objects option and enter the number of days after which the files will be deleted.
  • Click on Create rule and the automatic file deletion rule will be applied.

Security, Access, Deployment, And Data Flows

Clone this wiki locally