-
Notifications
You must be signed in to change notification settings - Fork 164
[QC-326] Aggregators #488
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
[QC-326] Aggregators #488
Changes from all commits
Commits
Show all changes
41 commits
Select commit
Hold shift + click to select a range
fc75176
Add AggregatorInterface.
Barthelemy 3174965
AggregatorRunner et al. Topology ok
Barthelemy a05dace
organize includes
Barthelemy 18afdc2
introduction of Aggregator class to hold a user code instance
Barthelemy 14255ee
update doxy aggregators
Barthelemy 8e1185b
aggregator change
Barthelemy 8a035d5
continued
Barthelemy df98348
Get data in Aggregator
Barthelemy d130b05
fix comma
Barthelemy efbc01f
fix merge
Barthelemy bf392bb
format
Barthelemy 48b6b68
Aggregators.. aggregate
Barthelemy 41e21ad
add non working advanced topology
Barthelemy 91b8dee
add non working advanced topology
Barthelemy 3a3901f
Works with other aggregators as input
Barthelemy 66c3a96
format
Barthelemy 8f8fc1e
script to configure new aggregator
Barthelemy 0d7481c
doc
Barthelemy e7cbbf6
some tests
Barthelemy b9b91df
format
Barthelemy ac863f0
fix
Barthelemy 49fd6e6
advanced example and custom parameters
Barthelemy fba2b59
feature full
Barthelemy 0d196f1
format
Barthelemy cdba36a
format
Barthelemy 52bd4fd
final clean up
Barthelemy 6b28990
format
Barthelemy 7e34b78
missing changes for MySQL backend
Barthelemy a733e78
First batch of fixes following PR's comments
Barthelemy 41b69fa
format
Barthelemy f0d8b76
update doc, incl. config description
Barthelemy 1e710b5
extract generateAggregators
Barthelemy c0b149a
progress on the changes proposed in the PR
Barthelemy 1c4f4a6
remove the option for the list of QOs
Barthelemy fb94085
better output, simplifying the policy handling
Barthelemy fdc5a87
make the binding less circomvoluted
Barthelemy cb04296
format
Barthelemy bcbdcdd
revert binding name, broken. use better loop in aggregate
Barthelemy 4495651
Merge branch 'master' into aggregators
Barthelemy c220210
Fix for piece that got lost
Barthelemy c4d4c3e
remove the option 'all' for the Checks.
Barthelemy File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| { | ||
| "qc": { | ||
| "config": { | ||
| "database": { | ||
| "implementation": "CCDB", | ||
| "host": "ccdb-test.cern.ch:8080", | ||
| "username": "not_applicable", | ||
| "password": "not_applicable", | ||
| "name": "not_applicable" | ||
| }, | ||
| "Activity": { | ||
| "number": "42", | ||
| "type": "2" | ||
| }, | ||
| "monitoring": { | ||
| "url": "infologger:///debug?qc" | ||
| }, | ||
| "consul": { | ||
| "url": "http://consul-test.cern.ch:8500" | ||
| }, | ||
| "conditionDB": { | ||
| "url": "ccdb-test.cern.ch:8080" | ||
| } | ||
| }, | ||
| "tasks": { | ||
| "dataSizeTask": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonTask", | ||
| "moduleName": "QcSkeleton", | ||
| "detectorName": "TST", | ||
| "cycleDurationSeconds": "10", | ||
| "maxNumberCycles": "-1", | ||
| "dataSource": { | ||
| "type": "dataSamplingPolicy", | ||
| "name": "tst2" | ||
| }, | ||
| "location": "remote" | ||
| }, | ||
| "someNumbersTask": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonTask", | ||
| "moduleName": "QcSkeleton", | ||
| "detectorName": "TST", | ||
| "cycleDurationSeconds": "10", | ||
| "maxNumberCycles": "-1", | ||
| "dataSource": { | ||
| "type": "dataSamplingPolicy", | ||
| "name": "tst1" | ||
| }, | ||
| "location": "remote" | ||
| } | ||
| }, | ||
| "checks": { | ||
| "dataSizeCheck": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonCheck", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAny", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Task", | ||
| "name": "dataSizeTask", | ||
| "MOs": ["example"] | ||
| }] | ||
| }, | ||
| "someNumbersCheck": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonCheck", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnEachSeparately", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Task", | ||
| "name": "someNumbersTask", | ||
| "MOs": ["example"] | ||
| }] | ||
| } | ||
| }, | ||
| "aggregators": { | ||
| "MyAggregator1": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonAggregator", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAll", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Check", | ||
| "name": "dataSizeCheck" | ||
| }, { | ||
| "type": "Check", | ||
| "name": "someNumbersCheck" | ||
| }] | ||
| }, | ||
| "MyAggregator2": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonAggregator", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAll", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Check", | ||
| "name": "dataSizeCheck" | ||
| }, { | ||
| "type": "Check", "": "this one is using onEachSeparately and thus it sends under a full path", | ||
| "name": "someNumbersCheck", | ||
| "QOs": ["someNumbersTask/example"], "": "also possible to ignore it altogether, meaning we take all objects" | ||
| }] | ||
| }, | ||
| "MyAggregator3": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonAggregator", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAll", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Aggregator", | ||
| "name": "MyAggregator1", "": "no QOs parameter -> all QOs of this aggregator", | ||
| "QOs": ["newQuality"], "": "list all objects we are interested in" | ||
| }, { | ||
| "type": "Aggregator", | ||
| "name": "MyAggregator2", | ||
| "QOs": ["newQuality", "another"], "": "if we omitted the QOs for a data source we would default to OnAny" | ||
| }] | ||
| } | ||
| } | ||
| }, | ||
| "dataSamplingPolicies": [ | ||
| { | ||
| "id": "tst1", | ||
| "active": "true", | ||
| "machines": [], | ||
| "query" : "sum:TST/SUM/2;param:TST/PARAM/2", | ||
| "samplingConditions": [ | ||
| { | ||
| "condition": "random", | ||
| "fraction": "0.1", | ||
| "seed": "32112332123" | ||
| } | ||
| ], | ||
| "blocking": "false" | ||
| }, | ||
| { | ||
| "id": "tst2", | ||
| "active": "true", | ||
| "machines": [], | ||
| "query" : "data:TST/DATA", | ||
| "samplingConditions": [ | ||
| { | ||
| "condition": "payloadSize", | ||
| "lowerLimit": "8000", | ||
| "upperLimit": "10000" | ||
| } | ||
| ], | ||
| "blocking": "false" | ||
| } | ||
| ] | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,107 @@ | ||
| { | ||
| "qc": { | ||
| "config": { | ||
| "database": { | ||
| "implementation": "CCDB", | ||
| "host": "ccdb-test.cern.ch:8080", | ||
| "username": "not_applicable", | ||
| "password": "not_applicable", | ||
| "name": "not_applicable" | ||
| }, | ||
| "Activity": { | ||
| "number": "42", | ||
| "type": "2" | ||
| }, | ||
| "monitoring": { | ||
| "url": "infologger:///debug?qc" | ||
| }, | ||
| "consul": { | ||
| "url": "http://consul-test.cern.ch:8500" | ||
| }, | ||
| "conditionDB": { | ||
| "url": "ccdb-test.cern.ch:8080" | ||
| } | ||
| }, | ||
| "tasks": { | ||
| "QcTask": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonTask", | ||
| "moduleName": "QcSkeleton", | ||
| "detectorName": "TST", | ||
| "cycleDurationSeconds": "10", | ||
| "maxNumberCycles": "-1", | ||
| "dataSource_comment": "The other type of dataSource is \"direct\", see basic-no-sampling.json.", | ||
| "dataSource": { | ||
| "type": "dataSamplingPolicy", | ||
| "name": "tst-raw" | ||
| }, | ||
| "taskParameters": { | ||
| "myOwnKey": "myOwnValue" | ||
| }, | ||
| "location": "remote" | ||
| } | ||
| }, | ||
| "checks": { | ||
| "QcCheck": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonCheck", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAny", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Task", | ||
| "name": "QcTask", | ||
| "MOs": ["example"] | ||
| }] | ||
| }, | ||
| "QcCheck2": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonCheck", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnEachSeparately", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Task", | ||
| "name": "QcTask", | ||
| "MOs": ["example"] | ||
| }] | ||
| } | ||
| }, | ||
| "aggregators": { | ||
| "MyAggregator1": { | ||
| "active": "true", | ||
| "className": "o2::quality_control_modules::skeleton::SkeletonAggregator", | ||
| "moduleName": "QcSkeleton", | ||
| "policy": "OnAll", | ||
| "detectorName": "TST", | ||
| "dataSource": [{ | ||
| "type": "Check", "": "for a check which produces a single result do it like that", | ||
| "name": "QcCheck", "": "no need to specify each object because there is only 1" | ||
|
knopers8 marked this conversation as resolved.
|
||
| }, | ||
| { | ||
| "type": "Check", "": "for a check which produces a single result do it like that", | ||
| "name": "QcCheck2", "": "no need to specify each object because there is only 1" | ||
| }], | ||
| "aggregatorParameters": { | ||
| "myOwnKey": "myOwnValue" | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "dataSamplingPolicies": [ | ||
| { | ||
| "id": "tst-raw", | ||
| "active": "true", | ||
| "machines": [], | ||
| "query": "random:TST/RAWDATA/0", | ||
| "samplingConditions": [ | ||
| { | ||
| "condition": "random", | ||
| "fraction": "0.1", | ||
| "seed": "1234" | ||
| } | ||
| ], | ||
| "blocking": "false" | ||
| } | ||
| ] | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.