HRQB 12 - Setup QB API and Load tasks

[ghukill]

Purpose and background context

This PR adds a QBClient class (commit) that is designed to interact with the Quickbase (QB) API.

This client handles logistical things like API credentials, making requests, etc., but also includes some helper methods to query and upsert data to QB.

Example: when upserting data into QB, you must provide a QB Field ID instead of the Field name. These Field IDs are not obvious, and would be cumbersome to code against in the ETL work. This client allows for upserting data via the method upsert_records() that accepts a {Field name:value} structure, where it retrieves information about the table and converts the upload payload to {Field id:value} structure on-the-fly.

Currently there are a couple of additional methods to get information about the QB app, and otherwise describing QB Tables and Fields. It is anticipated as this hrqb-client develops, additional convenience methods will be added (e.g. querying records or displaying statistics) that may be directly accessble via CLI commands.

In summary, this QBClient will be the primary tool used by Load Tasks to upsert data into Quickbase.

These changes also introduce two new important env vars:

• QUICKBASE_API_TOKEN: API token used by this app to connect to QB
• QUICKBASE_APP_ID: identifier of the QB app

How can a reviewer manually see the effects of these changes?

Without any CLI commands yet, the best way to get a quick understanding of this client is an ipython shell.

Set env vars:

QUICKBASE_API_TOKEN=M<TOKEN HERE>
QUICKBASE_APP_ID=bt5hg8uz7

Start shell:

pipenv run ipython

Explore the client a bit:

from hrqb.utils.quickbase import QBClient

# instantiate client
qbclient = QBClient()

# get Quickbase app information
qbclient.get_app_info()

# get a pandas Dataframe of all Tables
 qbclient.get_tables()

# get Table ID from a name
qbclient.get_table_id("HRQBClient Messages")

# the QBClient caches requests when the path + parameters are identical
# this is helpful for methods that have to re-run queries to get Table and Field identifiers
# the following shows the keys in the cache dictionary, hinting at how it's stored and checked
# when new requests are made
qbclient._cache.keys()

# get all Fields for a Table
qbclient.get_table_fields(qbclient.get_table_id('HRQBClient Messages'))

# get a Field Name-to-ID map that is used by upserting methods
qbclient.get_table_fields_name_to_id(qbclient.get_table_id('HRQBClient Messages'))

# demonstrate how upserts are performed
# 1. create an upsert payload, using human-friendly Field names like 'Message'
upsert_payload = qbclient.prepare_upsert_payload(
    qbclient.get_table_id('HRQBClient Messages'),
    [
        {
            "Message":"Hello World!"
        }
    ],
    merge_field=None
)

# look at payload
display(upsert_payload)

#2. upsert into Table 'HRQBClient Messages'
# note the output, which is effectively the output artifact of a Load Task 
# when they are built in future PRs
qbclient.upsert_records(upsert_payload)

Includes new or updated dependencies?

YES

Changes expectations for external applications?

NO

What are the relevant tickets?

• https://mitlibraries.atlassian.net/browse/HRQB-12

Developer

• All new ENV is documented in README
• All new ENV has been added to staging and production environments
• All related Jira tickets are linked in commit message(s)
• Stakeholder approval has been confirmed (or is not needed)

Code Reviewer(s)

• The commit message is clear and follows our guidelines (not just this PR message)
• There are appropriate tests covering any new functionality
• The provided documentation is sufficient for understanding any new functionality introduced
• Any manual tests have been performed or provided examples verified
• New dependencies are appropriate or there were no changes

[ghukill]

@ehanson8 , @jonavellecuerdo - it just occurred to me, if you're digging into the QBClient for this PR, you may have the question (as I do):

If the QBClient caches API request/responses, what happens if you insert data and then want to retrieve the updated results from a Table?

The answer is: the QBClient does not yet retrieve actual records. When it does, it'll need some kind of cache clearing (or maybe as simple as some methods don't use the cache). Which is all to say, it'll likely get some more functionality as we progress, and the caching could be an area for slight revision if needed. But for the most part, the Tables and their Fields are quite static (particularly across a single invocation of the QBClient).

[ghukill]

@jonavellecuerdo - you had asked in the last PR where and when Task.table_name would be used.

While only used for QuickbaseUpsertTask at this time, here is an example of it becoming helpful as the Tasks start to take more shape: https://github.com/MITLibraries/hrqb-client/pull/8/files#diff-d86e429c2a92dbd803698c92d15054d84cd0886541acd33a95f26ec5e48d39e6R87.

[jonavellecuerdo]

@ghukill The QBClient module thus far is understandable and well-written! I just have a couple of clarification questions for you, but it's looking s'wonderful.

[jonavellecuerdo]

@ghukill Looks good to me! Just added one more comment re: including additional info in docstring.

[ehanson8]

Looks good, one suggestion for breaking up a large method