User stages

[bencodeorg]

This creates a analytics/Redshift-only table, user_stages, which aggregates student progress to the "stage" level (also known as a "lesson" externally in some of our courses). Our production data model includes student progress at the "level" granularity (user_levels) and the "script" granularity (user_scripts), but doesn't actively track anything in between.

This PR:

• creates a redshift_rollups cron task to run every playbook in the aws/redshift/playbooks directory, and adds a new playbook to create/refresh user_stages.

[bencodeorg]

Ah I thought the [ci skip] tag on the second commit would just prevent a circle build for the second commit (which is a SQL change with no production implications) instead of for the whole PR. The first modifies a cron job, so would actually be run via a production machine, but I think you've told me before @wjordan there's not really any testing that would be helpful there anyway?

[wjordan]

Is there a specific reason favoring adding a second playbook, versus adding an extra step to the existing playbook?

[wjordan]

the referenced tables/user_stages.sql seems to be missing

[bencodeorg]

D'oh. Added user_stages script.

Re: adding second playbook, there's no functional difference, just an organizational change. All the stuff in the first playbook relates explicitly to reporting we do to regional partners, and the new one for user stages is more broadly useful for analytical purposes.

My thought was that we might add additional playbooks in the future as we do more of this sort of stuff (I'd more descriptively call these "analytical aggregates"), and was trying to prevent a single playbook from getting unwieldy.

[bencodeorg]

@wjordan this look ok to you now?

[wjordan]

👍 looks great, sorry for losing track of this.