Skip to content

[QC-602] Batch processing in QC#779

Merged
Barthelemy merged 5 commits into
AliceO2Group:masterfrom
knopers8:merge-files-2
Jul 23, 2021
Merged

[QC-602] Batch processing in QC#779
Barthelemy merged 5 commits into
AliceO2Group:masterfrom
knopers8:merge-files-2

Conversation

@knopers8
Copy link
Copy Markdown
Collaborator

This allows to process partial data by QC Tasks and dump/merge results into a file.
Then, the merged results can be picked up by the last parts of the QC chain (Checks, Aggregators, storage).

This allows to process partial data by QC Tasks and dump/merge results into a file.
Then, the merged results can be picked up by the last parts of the QC chain (Checks, Aggregators, storage).
@knopers8 knopers8 requested a review from Barthelemy as a code owner July 22, 2021 14:28
@knopers8
Copy link
Copy Markdown
Collaborator Author

@Barthelemy This is essentially working and ready for a review, I only have to write tests now.
I am especially not sure about the naming, so any other suggestions for "local batch", "remote batch", "RootFileSink" and "RootFileSource" will be very welcome.

@knopers8 knopers8 changed the title [WIP] [QC-602] Batch processing in QC [QC-602] Batch processing in QC Jul 22, 2021
@Barthelemy
Copy link
Copy Markdown
Collaborator

I'll review it today.

@knopers8
Copy link
Copy Markdown
Collaborator Author

I am not sure it is related to tuning the test parameters #777
but three tests now failed in o2-dataflow:

## sw/BUILD/QualityControl-latest/log
--
 5/48 Test #38: testFactory .............................***Timeout  20.97 sec
--
 9/48 Test #16: testCcdbDatabase ........................***Timeout  30.37 sec
--
36/48 Test #32: multinode_test ..........................***Failed   42.98 sec
--
94% tests passed, 3 tests failed out of 48

@Barthelemy
Copy link
Copy Markdown
Collaborator

Weird. I did not touch testCheckWorkflow which is the only one I see actually failing.

The following tests FAILED:
	 25 - testCheckWorkflow (Timeout)
Errors while running CTest

I will also increase the timeout for testCheckWorkflow

Copy link
Copy Markdown
Collaborator

@Barthelemy Barthelemy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good, the naming is ok, I don't have a better proposal.

The infrastructure generator is what it is ... :)

I tried to run the test but I get:

Error, monitor object of the QC Task could not be found.

Comment thread Framework/script/o2-qc-batch-test.sh
@knopers8
Copy link
Copy Markdown
Collaborator Author

The test could fail this way if you provide it with wrong UNIQUE_ID env var. It passes CI now, so I doubt this is a real problem.

@Barthelemy Barthelemy merged commit ef1655d into AliceO2Group:master Jul 23, 2021
@knopers8 knopers8 deleted the merge-files-2 branch July 23, 2021 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants