chng failing with `assert len(filepaths_to_download) <= 6, "more files dropped than expected" AssertionError: more files dropped than expected`

[korlaxxalrok]

Actual Behavior:

source: chng

Per @jingjtang's investigation:

It seems that there are no counts delivered from 2021-12-23 to 2021-12-26.

Fails with:

Slack ref: https://delphi-org.slack.com/archives/C01LZ3A2UMU/p1640748872124100

❗ Indicator exception

Logger: delphi_changehc.run
Timestamp: 2021-12-29T03:34:02.771Z
Exception: Traceback (most recent call last):
File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
 return _run_code(code, main_globals, None,
File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
 exec(code, run_globals)
File "/mnt/data2/indicators/runtime/changehc/env/lib/python3.8/site-packages/delphi_utils/runner.py", line 64, in <module>
 run_indicator_pipeline(indicator_module.run.run_module,
File "/mnt/data2/indicators/runtime/changehc/env/lib/python3.8/site-packages/delphi_utils/runner.py", line 43, in run_indicator_pipeline
 indicator_fn(params)
File "/mnt/data2/indicators/runtime/changehc/delphi_changehc/run.py", line 129, in run_module
 file_dict = retrieve_files(params, filedate, logger)
File "/mnt/data2/indicators/runtime/changehc/delphi_changehc/run.py", line 29, in retrieve_files
 download_counts(filedate, params["indicator"]["input_cache_dir"], params["indicator"]["ftp_conn"])
File "/mnt/data2/indicators/runtime/changehc/delphi_changehc/download_ftp_files.py", line 64, in download_counts
 get_files_from_dir(sftp, filedate, out_path)
File "/mnt/data2/indicators/runtime/changehc/delphi_changehc/download_ftp_files.py", line 36, in get_files_from_dir
 assert len(filepaths_to_download) <= 6, "more files dropped than expected"
AssertionError: more files dropped than expected

Expected behavior

chng should not fail.

Context

This looks like provider issue. As this is happening during the Winter Holiday period we aren't meeting with them for a week or two.

@jingjtang has emailed CHNG 2022-01-03, waiting for CHNG to add those files to the SFTP server

[krivard]

This is actually caused by the new file for flu inpatient (which make more sense -- we wouldn't see "more files than expected" if files were actually missing).

We need to update the inequality in this test to be:

assert len(filepaths_to_download) <= 7

[krivard]

We'll also need to generate a data patch for:

• issue range: 20211222-20220105