[CTIS] Create new set of CID lists to send to FB

[nmdefries]

Description

Support creating and sharing new CID lists. This duplicates our existing CID lists (step 1/part a and full) under different names, and produces a new CID list that takes into account whether or not respondents completed the A/B modules and provided key demographic information. These new CID lists are considered experimental for now and are not reported to data users. They are saved to a new local directory and uploaded to a new directory on the FTP site.

Specs on CID list definitions and associated Asana task.

Changelog

• Makefile:
  • Add new experimental SFTP destination folder.
  • Define new post-experimental-cids target.
  • Add as a dependency wherever the original CIDs are required.
• contingency_variables.R: Remove age and gender definitions.
• variables.R: Add age and gender definitions now that they'll be created for both API and contingency pipelines.
• responses.R:
  • Move gender and age creation out of the contingency-only block.
  • Add filter_module_complete_responses() to create valid response lists for new CID list definition.
• run.R: Add new CID list creation and save steps.
• weights.R:
  • Generalize write_cid() with new args so provided data can be saved with different filename and in a different location. module_type, if provided, is added to the printed message.
  • Create generate_cid_list_filename() to remove from write_cid()
  • Create write_cid_experimental_wrapper() to pass new args to write_cid() as need to write CID lists with new names.
• params.json.template, params.json.production.template, integration-tests/.../params-full.json, integration-tests/.../params-test.json: Add experimental_weights_out_dir field to save experimental CID lists separately from other CID lists.
• integration-tests/.../helper-relativize.R: Relativize new experimental_weights_out_dir field.

[nmdefries]

We'll need to create a new SFTP directory fb-interchange/cmu_respondent_ww_ids to upload the new CID lists to.

[nmdefries]

@krivard Could you take a look at the Makefile changes?

[krivard]

Makefile changes look fine, with the caveat that if we ever need to add a third kind of CID files we'll want to refactor to reduce the amount of copy-paste. I'm not irked by 2 copies, but three would be too many.

[capnrefsmmat]

Looks fine on a skim. Has this been tested on a real Qualtrics data file to be sure the Finished stuff works fine?

[nmdefries]

Yep, I tested it out, seems to work.

The destination dir has been created, so this is ready to merge @krivard.

[krivard]

I'm doing a release today, so this will take effect tomorrow morning.