Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix logging spillover #2174

Merged
merged 31 commits into from
Nov 14, 2023
Merged
Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
c849b54
Update pre-commit hooks versions
vladsavelyev Oct 27, 2023
c41eb65
Fix
vladsavelyev Oct 27, 2023
b12d6d6
Ruff
vladsavelyev Oct 27, 2023
ae7194d
Update hooks and docs
vladsavelyev Oct 27, 2023
8bdbd8b
ruff --fix
vladsavelyev Oct 27, 2023
2eba2ed
Use and apply Ruff formatter
vladsavelyev Oct 27, 2023
06097f3
Use and apply Ruff formatter
vladsavelyev Oct 27, 2023
c70ed26
Fix further formatting
vladsavelyev Oct 27, 2023
2310e24
Fix further formatting - 2
vladsavelyev Oct 27, 2023
d937143
Merge branch 'master' into add-ruff-linter
vladsavelyev Oct 27, 2023
06b79a7
Merge branch 'master' into add-ruff-linter
vladsavelyev Nov 3, 2023
6ac0d3b
Format linting, remove OrderedDict
vladsavelyev Nov 4, 2023
405f036
Fix linting
vladsavelyev Nov 4, 2023
cbc0b80
Fix
vladsavelyev Nov 4, 2023
4564452
Fix
vladsavelyev Nov 4, 2023
783d99e
Clean up
vladsavelyev Nov 5, 2023
ad5c68a
Fix label on Per-Sequence GC Content
vladsavelyev Oct 20, 2023
71f3b5b
Merge branch 'master' into add-ruff-linter
vladsavelyev Nov 6, 2023
2a41b9d
Merge branch 'master' into add-ruff-linter
vladsavelyev Nov 10, 2023
833757b
Fix linting
vladsavelyev Nov 10, 2023
87347cf
Merge branch 'master' into add-ruff-linter
vladsavelyev Nov 13, 2023
c002ec1
Linting
vladsavelyev Nov 13, 2023
87a68e9
Remove docs/modules/sentieon.md
vladsavelyev Nov 13, 2023
2645973
Merge branch 'master' into add-ruff-linter
vladsavelyev Nov 13, 2023
1b4e533
Fix
vladsavelyev Nov 13, 2023
a94ef8c
Remove ordered dict
vladsavelyev Nov 14, 2023
c339779
Fix bug in custom content
vladsavelyev Nov 14, 2023
92feb9f
Add r specifiers to regex strings to avoid warnings. Use logger vs lo…
vladsavelyev Nov 14, 2023
05e4fd9
More regex escaping
vladsavelyev Nov 14, 2023
19316bb
[automated] Update CHANGELOG.md
multiqc-bot Nov 14, 2023
92ffaf1
Merge branch 'master' into fix-logging-spill
vladsavelyev Nov 14, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
11 changes: 6 additions & 5 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,9 @@ repos:
rev: v0.1.3
hooks:
- id: ruff-format
# - repo: https://github.com/astral-sh/ruff-pre-commit
# rev: v0.1.3
# hooks:
# - id: ruff
# args: [--fix, --exit-non-zero-on-fix]

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.1.3
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ Highlights:
- Software versions: allow any string as a version tag ([#2166](https://github.com/ewels/MultiQC/pull/2166))
- Remove position:absolute from table values ([#2169](https://github.com/ewels/MultiQC/pull/2169))
- Fix custom anchors for kraken ([#2170](https://github.com/ewels/MultiQC/pull/2170))
- Fix logging spillover ([#2174](https://github.com/ewels/MultiQC/pull/2174))

### New Modules

Expand Down
35 changes: 18 additions & 17 deletions docs/core/development/modules.md
Original file line number Diff line number Diff line change
Expand Up @@ -500,8 +500,8 @@ instead:
```python
for f in self.find_log_files('mymod', filehandles=True):
# f['f'] is now a filehandle instead of contents
for l in f['f']:
print( l )
for line in f['f']:
print(line)
```

This is good if the file is large, as Python doesn't read the entire
Expand All @@ -525,8 +525,8 @@ class MultiqcModule(BaseMultiqcModule):

def parse_logs(self, f):
data = {}
for l in f.splitlines():
s = l.split()
for line in f.splitlines():
s = line.split()
data[s[0]] = s[1]
return data
```
Expand Down Expand Up @@ -764,19 +764,20 @@ To give more informative table headers and configure things like
data scales and colour schemes, you can supply an extra dict:

```python
headers = OrderedDict()
headers['first_col'] = {
'title': 'First',
'description': 'My First Column',
'scale': 'RdYlGn-rev'
}
headers['second_col'] = {
'title': 'Second',
'description': 'My Second Column',
'max': 100,
'min': 0,
'scale': 'Blues',
'suffix': '%'
headers = {
'first_col': {
'title': 'First',
'description': 'My First Column',
'scale': 'RdYlGn-rev'
},
'second_col': {
'title': 'Second',
'description': 'My Second Column',
'max': 100,
'min': 0,
'scale': 'Blues',
'suffix': '%'
}
}
self.general_stats_addcols(data, headers)
```
Expand Down
100 changes: 52 additions & 48 deletions docs/core/development/plots.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ description: Visualising your data
# Plotting Functions

MultiQC plotting functions are held within `multiqc.plots` submodules.
To use them, simply import the modules you want, eg.:
To use them, simply import the modules you want, e.g.:

```python
from multiqc.plots import bargraph, linegraph
Expand Down Expand Up @@ -80,18 +80,18 @@ html_content = bargraph.plot(data, cats)
```

If `cats` is given as a dict instead of a list, you can specify a nice name
and a colour too. Make it an OrderedDict to specify the order:
and a colour too:

```python
from collections import OrderedDict
cats = OrderedDict()
cats['aligned'] = {
'name': 'Aligned Reads',
'color': '#8bbc21'
}
cats['not_aligned'] = {
'name': 'Unaligned Reads',
'color': '#f7a35c'
cats = {
"aligned": {
'name': 'Aligned Reads',
'color': '#8bbc21'
},
"not_aligned": {
'name': 'Unaligned Reads',
'color': '#f7a35c'
}
}
```

Expand Down Expand Up @@ -120,7 +120,7 @@ config = {
'yCeiling': None, # Maximum value for automatic axis limit (good for percentages)
'yFloor': None, # Minimum value for automatic axis limit
'yMinRange': None, # Minimum range for axis
'yDecimals': True, # Set to false to only show integer labels
'yDecimals': True, # Set to False to only show integer labels
'ylab_format': None, # Format string for x axis labels. Defaults to {value}
'stacking': 'normal', # Set to None to have category bars side by side
'use_legend': True, # Show / hide the legend
Expand Down Expand Up @@ -187,12 +187,16 @@ cats = [
Or with additional customisation such as name and colour:

```python
from collections import OrderedDict
cats = [OrderedDict(), OrderedDict()]
cats[0]['aligned_reads'] = {'name': 'Aligned Reads', 'color': '#8bbc21'}
cats[0]['unaligned_reads'] = {'name': 'Unaligned Reads', 'color': '#f7a35c'}
cats[1]['aligned_base_pairs'] = {'name': 'Aligned Base Pairs', 'color': '#8bbc21'}
cats[1]['unaligned_base_pairs'] = {'name': 'Unaligned Base Pairs', 'color': '#f7a35c'}
cats = [
{
"aligned_reads": {"name": "Aligned Reads", "color": "#8bbc21"},
"unaligned_reads": {"name": "Unaligned Reads", "color": "#f7a35c"},
},
{
"aligned_base_pairs": {"name": "Aligned Base Pairs", "color": "#8bbc21"},
"unaligned_base_pairs": {"name": "Unaligned Base Pairs", "color": "#f7a35c"},
},
]
html_content = bargraph.plot([data, data], cats, config)
```

Expand Down Expand Up @@ -255,27 +259,27 @@ config = {
'xmax': None, # Max x limit
'xmin': None, # Min x limit
'xLog': False, # Use log10 x axis?
'xDecimals': True, # Set to false to only show integer labels
'xDecimals': True, # Set to False to only show integer labels
'yCeiling': None, # Maximum value for automatic axis limit (good for percentages)
'yFloor': None, # Minimum value for automatic axis limit
'yMinRange': None, # Minimum range for axis
'ymax': None, # Max y limit
'ymin': None, # Min y limit
'yLog': False, # Use log10 y axis?
'yDecimals': True, # Set to false to only show integer labels
'yDecimals': True, # Set to False to only show integer labels
'yPlotBands': None, # Highlighted background bands. See http://api.highcharts.com/highcharts#yAxis.plotBands
'xPlotBands': None, # Highlighted background bands. See http://api.highcharts.com/highcharts#xAxis.plotBands
'yPlotLines': None, # Highlighted background lines. See http://api.highcharts.com/highcharts#yAxis.plotLines
'xPlotLines': None, # Highlighted background lines. See http://api.highcharts.com/highcharts#xAxis.plotLines
'xLabelFormat': '{value}', # Format string for the axis labels
'yLabelFormat': '{value}', # Format string for the axis labels
'tt_label': '{point.x}: {point.y:.2f}', # Use to customise tooltip label, eg. '{point.x} base pairs'
'tt_label': '{point.x}: {point.y:.2f}', # Use to customise tooltip label, e.g. '{point.x} base pairs'
'tt_decimals': None, # Tooltip decimals when categories = True (when false use tt_label)
'tt_suffix': None, # Tooltip suffix when categories = True (when false use tt_label)
'pointFormat': None, # Replace the default HTML for the entire tooltip label
'click_func': function(){}, # Javascript function to be called when a point is clicked
'cursor': None # CSS mouse cursor type. Defaults to pointer when 'click_func' specified
'reversedStacks': False # Reverse the order of the category stacks. Defaults True for plots with Log10 option
'cursor': None, # CSS mouse cursor type. Defaults to pointer when 'click_func' specified
'reversedStacks': False, # Reverse the order of the category stacks. Defaults True for plots with Log10 option
'height': 512 # The default height of the plot, in pixels
}
html_content = linegraph.plot(data, config)
Expand Down Expand Up @@ -422,11 +426,10 @@ should be displayed in.

For more customisation, the headers can be supplied as a dictionary. Each
key should match the keys used in the data dictionary, but values can
customise the output. If you want to specify the order of the columns, you
must use an `OrderedDict`.
customise the output.

Finally, the function accepts a config dictionary as a third parameter.
This can set global options for the table (eg. a title) and can also hold
This can set global options for the table (e.g. a title) and can also hold
default values to customise the output of all table columns.

The default header keys are:
Expand All @@ -444,7 +447,7 @@ single_header = {
'scale': 'GnBu', # Colour scale for colour coding. False to disable.
'bgcols': None, # Dict with values: background colours for categorical data.
'colour': '<auto>', # Colour for column grouping
'suffix': None, # Suffix for value (eg. '%')
'suffix': None, # Suffix for value (e.g. '%')
'format': '{:,.1f}', # Value format string - default 1 decimal place
'cond_formatting_rules': None, # Rules for conditional formatting table cell values - see docs below
'cond_formatting_colours': None, # Styles for conditional formatting of table cell values
Expand Down Expand Up @@ -491,7 +494,7 @@ table_html = table.plot(data)
```

A more complicated version with ordered columns, defaults and column-specific
settings (eg. no decimal places):
settings (e.g. no decimal places):

```python
data = {
Expand All @@ -506,24 +509,25 @@ data = {
'aligned_percent': 14.820411484
}
}
headers = OrderedDict()
headers['aligned_percent'] = {
'title': '% Aligned',
'description': 'Percentage of reads that aligned',
'suffix': '%',
'max': 100,
'format': '{:,.0f}' # No decimal places please
}
headers['aligned'] = {
'title': '{} Aligned'.format(config.read_count_prefix),
'description': 'Aligned Reads ({})'.format(config.read_count_desc),
'shared_key': 'read_count',
'modify': lambda x: x * config.read_count_multiplier
}
config = {
'namespace': 'My Module',
'min': 0,
'scale': 'GnBu'
headers = {
"aligned_percent": {
'title': '% Aligned',
'description': 'Percentage of reads that aligned',
'suffix': '%',
'max': 100,
'format': '{:,.0f}' # No decimal places please
},
"aligned": {
'title': '{} Aligned'.format(config.read_count_prefix),
'description': 'Aligned Reads ({})'.format(config.read_count_desc),
'shared_key': 'read_count',
'modify': lambda x: x * config.read_count_multiplier
},
"config": {
'namespace': 'My Module',
'min': 0,
'scale': 'GnBu'
}
}
table_html = table.plot(data, headers, config)
```
Expand Down Expand Up @@ -777,7 +781,7 @@ config = {
yDecimals: true, // Set to false to only show integer labels
yPlotBands: undefined, // Highlighted background bands. See http://api.highcharts.com/highcharts#yAxis.plotBands
xPlotBands: undefined, // Highlighted background bands. See http://api.highcharts.com/highcharts#xAxis.plotBands
tt_label: "{point.x}: {point.y:.2f}", // Use to customise tooltip label, eg. '{point.x} base pairs'
tt_label: "{point.x}: {point.y:.2f}", // Use to customise tooltip label, e.g. '{point.x} base pairs'
pointFormat: undefined, // Replace the default HTML for the entire tooltip label
click_func: function () {}, // Javascript function to be called when a point is clicked
cursor: undefined, // CSS mouse cursor type. Defaults to pointer when 'click_func' specified
Expand Down Expand Up @@ -974,7 +978,7 @@ $("#YOUR_PLOT_ID").on("mqc_original_series_click", function (e, name) {

$("#YOUR_PLOT_ID").on("mqc_original_chg_source", function (e, name) {
// A plot with original images has had a request to change the
// original image source (eg. pressing Prev / Next)
// original image source (e.g. pressing Prev / Next)
});

$("#YOUR_PLOT_ID").on("mqc_plotexport_image", function (e, cfg) {
Expand Down
21 changes: 0 additions & 21 deletions docs/modules/sentieon.md

This file was deleted.

2 changes: 2 additions & 0 deletions multiqc/modules/adapterRemoval/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
from .adapterRemoval import MultiqcModule

__all__ = ["MultiqcModule"]
39 changes: 19 additions & 20 deletions multiqc/modules/adapterRemoval/adapterRemoval.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@

import logging
import re
from collections import OrderedDict

from multiqc import config
from multiqc.modules.base_module import BaseMultiqcModule, ModuleNoSamplesFound
Expand Down Expand Up @@ -268,23 +267,24 @@ def set_len_dist(self, len_dist_data):
self.len_dist_plot_data["all"][self.s_name][l_data[0]] = l_data[7]

def adapter_removal_stats_table(self):
headers = OrderedDict()
headers["percent_aligned"] = {
"title": "% Trimmed",
"description": "% trimmed reads",
"max": 100,
"min": 0,
"suffix": "%",
"scale": "RdYlGn-rev",
"shared_key": "percent_aligned",
}
headers["aligned_total"] = {
"title": "{} Reads Trimmed".format(config.read_count_prefix),
"description": "Total trimmed reads ({})".format(config.read_count_desc),
"modify": lambda x: x * config.read_count_multiplier,
"min": 0,
"scale": "PuBu",
"shared_key": "read_count",
headers = {
"percent_aligned": {
"title": "% Trimmed",
"description": "% trimmed reads",
"max": 100,
"min": 0,
"suffix": "%",
"scale": "RdYlGn-rev",
"shared_key": "percent_aligned",
},
"aligned_total": {
"title": "{} Reads Trimmed".format(config.read_count_prefix),
"description": "Total trimmed reads ({})".format(config.read_count_desc),
"modify": lambda x: x * config.read_count_multiplier,
"min": 0,
"scale": "PuBu",
"shared_key": "read_count",
},
}
if self.__any_collapsed:
headers["percent_collapsed"] = {
Expand Down Expand Up @@ -316,8 +316,7 @@ def adapter_removal_retained_chart(self):
"cpswitch_counts_label": "Number of Reads",
}

cats_pec = OrderedDict()

cats_pec = {}
if self.__any_paired:
cats_pec["paired_reads"] = {"name": "Uncollapsed Paired Reads"}

Expand Down
2 changes: 2 additions & 0 deletions multiqc/modules/afterqc/__init__.py
Original file line number Diff line number Diff line change
@@ -1 +1,3 @@
from .afterqc import MultiqcModule

__all__ = ["MultiqcModule"]