Python package errors during setup

[rogerthatdev]

from @glasnt feedback

Problem

When setup.sh calls seed_test_approver.py there is a python import error that causes setup.sh to fail

Steps to Reproduce

Steps to reproduce the behavior:

1. Run through setup.sh until 'Seeding default content..'

Expected behavior

No import errors.

Additional context

ImportError: cannot import name 'firestore' from 'google.cloud' (unknown location)
Exited [1] at line 118 setup.sh:
cat: setup.sh: No such file or directory

User environment:

• OS: cloud shell

Proposal

Replace python used in setup with bash gcloud CLI commands

[engelke]

Using bash won't help in this case, because actually interacting with Firestore is going to require code using a Cloud client of some kind. I can think of three approaches:

Option 1 - It might as well be Python. We can either have the setup script install google-cloud-firestore in the user's default Python environment, which would be rude, or create and use a virtual environment with it installed.

Option 2 - Use the gcloud CLI which can import data into Firestore from a Cloud Storage bucket.

Option 3 - Use Terraform to create Firestore documents one at a time, or load from a text file. One at a time would work well for the approver seeding, not so much for the rest of the data.

This is going to need discussion and a decision record.

[rogerthatdev]

Option 3 - Use Terraform to create Firestore documents one at a time, or load from a text file. One at a time would work well for the approver seeding, not so much for the rest of the data.

I took a look at what the possible solution here looks like and submitted PR #644 to start. Here's a break down of the changes we'll need to submit to replace the python seeding scripts with Terraform resources:

• Add tf resource for 'approver' document in Firestore
  • This is accomplished with the google_firestore_document resource, which will create the collection and add the document. The field parameter takes a json string of the Firestore projects.databases.documents fields field, which can be provided with a Terraform template data source (template_file) referencing a local template file (*.tftpl) that allows for using a new Terraform variable for approver_email. Additionally, a random_string resource is needed to provide document id.
  • Resolved via refactor: Terraform to seed approver and donors test data to Firestore #644
• Add tf google_firestore_document resource for each collection in test data
  • Similar to the approver collection, each collection (donors, campaigns, causes, donations) will need a template_file data source pointing to local template files for their field parameter. They will also need a corresponding json file with the test data, which will be looped using a count parameter in the corresponding google_firestore_document resource. The template file vars are populated using the jsondecode() function on the test data json file.
  • Resolved via refactor: Terraform to seed causes and donations test data to Firestore #650 refactor: Terraform to seed campaign test data to Firestore #647 refactor: Terraform to seed approver and donors test data to Firestore #644
• Include a seed_test_data toggle in the emblem_app module
  • To make the seeding of test data optional, there should be terraform variable that will be used to indicate whether or not to deploy those resources. It should be set to false by default
  • Resolved via refactor: Terraform to seed approver and donors test data to Firestore #644
• Update setup.sh to replace python invocation with tf seed_test_data variable
  • When the tf resources are in place, the setup script must be updated to use the SKIP_SEEDING environment variable to control the value for the tf variable seed_test_data. It must also prompt for an email to use for an approver, to pass along to the emblem-module as approver_email
  • Seed Firestore sample data with Terraform app module #651

Other notes:

• Because the creation of these resources can be toggled by the seed_test_data tf var, there is no need to make this one big PR merge. The work can be broken down so long as the resources added to Terraform include a count parameter with a conditional expression using seed_test_data that is set to false by default.
• Terraform will deploy a new resource for each document in every collection, therefore the Terraform plan and apply output will include a very high number of resources.
• While testing the creation of the donor documents went well, there was at least one test where it ran into a concurrency error from Firestore after creating 100 documents. I have not been able to consistently recreate this error, but warrants further investigation.

[rogerthatdev]

Due to the concurrency errors that occur when Terraform creates a large amount of Firestore documents, the Terraform solution may not be the best one at this time.