Optimize parsing of 2D data dictionaries in multiqc.utils.utils_functions.write_data_file()

[sstrong99]

• This comment contains a description of changes (with reason)
• CHANGELOG.md has been updated

Ran into an issue with multiqc hanging with large preseq files (10k lines). It seems to be this double loop over the 300 samples x 10000 lines that is causing the hang. I did some benchmarking with the below code. The results are

• the original method test_orig() took 5.7s
• test_opt took 0.4s, 14x faster, but only works if you can trust that all the data are the same type
• test_opt_check_al() took 3s, 1.9x faster

This PR currently implements test_opt, which requires that you trust that all the data are the same type. I'm not sure if that's an OK assumption.

def test_orig(data):
    return list(
        dict.fromkeys(
            [
                str(data_header)
                for sample_data in data.values()
                for data_header in sample_data.keys()
                if type(sample_data[data_header]) is not dict
            ]
        )
    )
​
def test_opt(data):
    h = set()
    for values in data.values():
        first_sub_value = next(iter(values))
        if isinstance(first_sub_value, dict):
            continue
        h |= values.keys()
    h = [str(item) for item in h]
    return h
​
def test_opt_check_all(data):
    h = set()
    for values in data.values():
        passing = {k for k, v in values.items() if not isinstance(v, dict)}
        h |= passing
    h = [str(item) for item in h]
    return h
​
# COMMAND ----------
​
import timeit
import string
​
test_dict={k: {k1: None for k1 in range(10000)} for k in list(string.ascii_lowercase)*12}
​
# COMMAND ----------
​
timeit.timeit(lambda: test_orig(test_dict), number=50)
​
# COMMAND ----------
​
timeit.timeit(lambda: test_opt(test_dict), number=50)
​
# COMMAND ----------
​
timeit.timeit(lambda: test_opt_check_all(test_dict), number=50)
​
# COMMAND ----------
​
opt_result = test_opt(test_dict)
orig_result = test_orig(test_dict)
opt_check_all_result = test_opt_check_all(test_dict)
assert opt_result == orig_result
assert opt_check_all_result == orig_result

[vladsavelyev]

Thank you for optimisation and benchmarking, that's really great!