Failure to write report when a table has an empty column that contains a modify lambda

[mjhipp]

Description of bug

This error occurs with the newest dev version (8851655 and before), potentially related to Pydantic, as it seems similar to #2455. This error does not occur in v1.21.

When running MultiQC and creating a table with an empty column that contains a modifier (lambda), the program fails. It fails at the end, after running modules.

This case can happen when a value may or may not be present in an input log file. I came across it on my custom plugin, but I created a simple custom_content file to replicate it. I attached both a file that works (foo) and a file that breaks (foo-missing).

File that triggers the error

This file succeeds:

foo_mqc.json

This file fails:

foo_missing_mqc.json

MultiQC Error log

❯ multiqc foo --verbose         

/// https://multiqc.info v1.22.dev0

[2024-05-14 17:02:00] multiqc.core.update_config                         [DEBUG  ]  This is MultiQC v1.22.dev0
[2024-05-14 17:02:00] multiqc.core.update_config                         [DEBUG  ]  Running Python 3.9.18 (main, Aug 24 2023, 18:16:58)  [Clang 15.0.0 (clang-1500.1.0.2.5)]
[2024-05-14 17:02:00] urllib3.connectionpool                             [DEBUG  ]  Starting new HTTPS connection (1): api.multiqc.info:443
[2024-05-14 17:02:00] urllib3.connectionpool                             [DEBUG  ]  https://api.multiqc.info:443 "GET /version?version_multiqc=1.22.dev0&version_python=3.9.18&operating_system=Darwin&is_docker=False&is_singularity=False&is_conda=False&is_ci=False&is_notebook=False HTTP/1.1" 200 178
[2024-05-14 17:02:00] multiqc.core.version_check                         [DEBUG  ]  Latest MultiQC version is v1.21, released 2024-02-28
[2024-05-14 17:02:00] multiqc.multiqc                                    [DEBUG  ]  Working dir : /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS
[2024-05-14 17:02:00] multiqc.multiqc                                    [DEBUG  ]  Template    : default
[2024-05-14 17:02:00] multiqc.multiqc                                    [DEBUG  ]  Command used: /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/bin/multiqc foo --verbose
[2024-05-14 17:02:00] multiqc.core.file_search                           [DEBUG  ]  Analysing modules: custom_content, ccs, ngsderive, purple, conpair, isoseq, lima, peddy, somalier, methylqa, mosdepth, phantompeakqualtools, qualimap, bamdst, preseq, hifiasm, quast, qorts, rna_seqc, rockhopper, rsem, rseqc, busco, bustools, goleft_indexcov, gffcompare, disambiguate, supernova, deeptools, sargasso, verifybamid, mirtrace, happy, mirtop, sambamba, gopeaks, homer, hops, macs2, theta2, snpeff, gatk, htseq, bcftools, featureCounts, fgbio, dragen, dragen_fastqc, dedup, pbmarkdup, damageprofiler, mapdamage, biobambam2, jcvi, mtnucratio, picard, vep, bakta, prokka, qc3C, nanostat, samblaster, samtools, sexdeterrmine, seqera_cli, eigenstratdatabasetools, bamtools, jellyfish, vcftools, longranger, stacks, varscan2, snippy, umitools, truvari, megahit, bbmap, bismark, biscuit, diamond, hicexplorer, hicup, hicpro, salmon, kallisto, slamdunk, star, hisat2, tophat, bowtie2, bowtie1, hostile, cellranger, snpsplit, odgi, pangolin, nextclade, freyja, humid, kat, leehom, librarian, nonpareil, adapterremoval, bbduk, clipandmerge, cutadapt, flexbar, sourmash, kaiju, bracken, kraken, malt, motus, trimmomatic, sickle, skewer, sortmerna, biobloomtools, fastq_screen, afterqc, fastp, fastqc, sequali, filtlong, prinseqplusplus, pychopper, porechop, pycoqc, minionqc, anglerfish, multivcfanalyzer, clusterflow, checkqc, bcl2fastq, bclconvert, interop, ivar, flash, seqyclean, optitype, whatshap, spaceranger, xenome, xengsort, metaphlan, seqwho
[2024-05-14 17:02:00] multiqc.core.file_search                           [INFO   ]  Search path: /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/foo
[2024-05-14 17:02:00] multiqc.report                                     [DEBUG  ]  Ignored 1 search patterns as didn't match running modules.
        searching | ████████████████████████████████████████ 100% 2/2
[2024-05-14 17:02:01] multiqc.report                                     [DEBUG  ]  Summary of files that were skipped by the search: |skipped_ignore_pattern: 1|, |skipped_no_match: 1|
[2024-05-14 17:02:01] multiqc.core.exec_modules                          [INFO   ]  Running module: custom_content
[2024-05-14 17:02:01] multiqc.modules.custom_content.custom_content      [INFO   ]  custom_data_lineplot: Found 1 samples (table)
[2024-05-14 17:02:01] multiqc.core.software_versions                     [DEBUG  ]  Reading software versions from config.software_versions
[2024-05-14 17:02:01] multiqc.core.write_results                         [INFO   ]  Existing reports found, adding suffix to filenames. Use '--force' to overwrite.
[2024-05-14 17:02:01] multiqc.core.write_results                         [DEBUG  ]  Moving data file from '/var/folders/yh/bxn3zgy56xvfdsw082bf7r3w0000gq/T/tmpdbd8ohm1/multiqc_data' to '/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/multiqc_data_6'
[2024-05-14 17:02:01] multiqc.core.write_results                         [INFO   ]  Data        : multiqc_data_6
[2024-05-14 17:02:01] multiqc.core.write_results                         [DEBUG  ]  Compressing plot data
[2024-05-14 17:02:01] multiqc.core.write_results                         [INFO   ]  Report      : multiqc_report_6.html
[2024-05-14 17:02:01] multiqc.core.write_results                         [DEBUG  ]  Full report path: /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/multiqc_report_6.html
[2024-05-14 17:02:01] multiqc.multiqc                                    [INFO   ]  MultiQC complete


❯ multiqc foo-missing --verbose         

/// https://multiqc.info v1.22.dev0

[2024-05-14 17:03:50] multiqc.core.update_config                         [DEBUG  ]  This is MultiQC v1.22.dev0
[2024-05-14 17:03:50] multiqc.core.update_config                         [DEBUG  ]  Running Python 3.9.18 (main, Aug 24 2023, 18:16:58)  [Clang 15.0.0 (clang-1500.1.0.2.5)]
[2024-05-14 17:03:50] urllib3.connectionpool                             [DEBUG  ]  Starting new HTTPS connection (1): api.multiqc.info:443
[2024-05-14 17:03:50] urllib3.connectionpool                             [DEBUG  ]  https://api.multiqc.info:443 "GET /version?version_multiqc=1.22.dev0&version_python=3.9.18&operating_system=Darwin&is_docker=False&is_singularity=False&is_conda=False&is_ci=False&is_notebook=False HTTP/1.1" 200 178
[2024-05-14 17:03:50] multiqc.core.version_check                         [DEBUG  ]  Latest MultiQC version is v1.21, released 2024-02-28
[2024-05-14 17:03:50] multiqc.multiqc                                    [DEBUG  ]  Working dir : /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS
[2024-05-14 17:03:50] multiqc.multiqc                                    [DEBUG  ]  Template    : default
[2024-05-14 17:03:50] multiqc.multiqc                                    [DEBUG  ]  Command used: /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/bin/multiqc foo-missing --verbose
[2024-05-14 17:03:50] multiqc.core.file_search                           [DEBUG  ]  Analysing modules: custom_content, ccs, ngsderive, purple, conpair, isoseq, lima, peddy, somalier, methylqa, mosdepth, phantompeakqualtools, qualimap, bamdst, preseq, hifiasm, quast, qorts, rna_seqc, rockhopper, rsem, rseqc, busco, bustools, goleft_indexcov, gffcompare, disambiguate, supernova, deeptools, sargasso, verifybamid, mirtrace, happy, mirtop, sambamba, gopeaks, homer, hops, macs2, theta2, snpeff, gatk, htseq, bcftools, featureCounts, fgbio, dragen, dragen_fastqc, dedup, pbmarkdup, damageprofiler, mapdamage, biobambam2, jcvi, mtnucratio, picard, vep, bakta, prokka, qc3C, nanostat, samblaster, samtools, sexdeterrmine, seqera_cli, eigenstratdatabasetools, bamtools, jellyfish, vcftools, longranger, stacks, varscan2, snippy, umitools, truvari, megahit, bbmap, bismark, biscuit, diamond, hicexplorer, hicup, hicpro, salmon, kallisto, slamdunk, star, hisat2, tophat, bowtie2, bowtie1, hostile, cellranger, snpsplit, odgi, pangolin, nextclade, freyja, humid, kat, leehom, librarian, nonpareil, adapterremoval, bbduk, clipandmerge, cutadapt, flexbar, sourmash, kaiju, bracken, kraken, malt, motus, trimmomatic, sickle, skewer, sortmerna, biobloomtools, fastq_screen, afterqc, fastp, fastqc, sequali, filtlong, prinseqplusplus, pychopper, porechop, pycoqc, minionqc, anglerfish, multivcfanalyzer, clusterflow, checkqc, bcl2fastq, bclconvert, interop, ivar, flash, seqyclean, optitype, whatshap, spaceranger, xenome, xengsort, metaphlan, seqwho
[2024-05-14 17:03:50] multiqc.core.file_search                           [INFO   ]  Search path: /Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/foo-missing
[2024-05-14 17:03:50] multiqc.report                                     [DEBUG  ]  Ignored 1 search patterns as didn't match running modules.
        searching | ████████████████████████████████████████ 100% 1/1
[2024-05-14 17:03:50] multiqc.core.exec_modules                          [INFO   ]  Running module: custom_content
[2024-05-14 17:03:50] multiqc.plots.plotly.violin                        [DEBUG  ]  No non-empty values found for metric: Percent
[2024-05-14 17:03:50] multiqc.modules.custom_content.custom_content      [INFO   ]  custom_data_lineplot: Found 1 samples (table)
[2024-05-14 17:03:50] multiqc.core.software_versions                     [DEBUG  ]  Reading software versions from config.software_versions
Traceback (most recent call last):
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/bin/multiqc", line 8, in <module>
    sys.exit(run_multiqc())
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/__main__.py", line 21, in run_multiqc
    multiqc.run_cli(prog_name="multiqc")
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/rich_click/rich_command.py", line 152, in main
    rv = self.invoke(ctx)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/multiqc.py", line 448, in run_cli
    result = run(*analysis_dir, clean_up=clean_up, cfg=cfg)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/multiqc.py", line 514, in run
    write_results()
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/core/write_results.py", line 42, in write_results
    render_and_export_plots()
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/core/write_results.py", line 313, in render_and_export_plots
    s.plot = plot.add_to_report(report)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/plots/plotly/violin.py", line 586, in add_to_report
    table_html, configuration_modal = make_table(self.main_table_dt, violin_id=self.id)
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/multiqc/plots/plotly/table.py", line 242, in make_table
    if header.get("hidden", False) is True:
  File "/Users/mikehipp/git-repos/twinstrandbio/MultiQC-DS/venv/lib/python3.9/site-packages/pydantic/main.py", line 811, in __getattr__
    raise AttributeError(f'{type(self).__name__!r} object has no attribute {item!r}')
AttributeError: 'TableColumn' object has no attribute 'get'

Before submitting

• I have read the troubleshooting documentation.
• I am using the latest release of MultiQC.
• I have included a full MultiQC log, not truncated.
• I have attached an input file (.zip if necessary) that triggers the error.

[vladsavelyev]

Sorry about this issue! Fixed with #2599, we'll release a patch release soon.