-
Notifications
You must be signed in to change notification settings - Fork 164
[QC-592] A facility to store ROOT objects from files to QCDB #862
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| // Copyright 2019-2020 CERN and copyright holders of ALICE O2. | ||
| // See https://alice-o2.web.cern.ch/copyright for details of the copyright holders. | ||
| // All rights not expressly granted are reserved. | ||
| // | ||
| // This software is distributed under the terms of the GNU General Public | ||
| // License v3 (GPL Version 3), copied verbatim in the file "COPYING". | ||
| // | ||
| // In applying this license CERN does not waive the privileges and immunities | ||
| // granted to it by virtue of its status as an Intergovernmental Organization | ||
| // or submit itself to any jurisdiction. | ||
|
|
||
| /// | ||
| /// \file runUploadRootObjects.cxx | ||
| /// \author Piotr Konopka | ||
| /// | ||
| /// \brief This is an executable which reads ROOT files with objects and puts them to QCDB. | ||
| /// | ||
| /// This is an executable which reads QAResults.root generated by DPL analysis tasks and puts them to QCDB. | ||
| /// It will ignore the directory structure and put all objects in under the task name specified as the argument. | ||
| /// By default the current date and time will be used as the start of validity, and the object will be valid for 10 years. | ||
|
|
||
| #include "QualityControl/QcInfoLogger.h" | ||
| #include "QualityControl/CcdbDatabase.h" | ||
| #include "QualityControl/MonitorObject.h" | ||
|
|
||
| #include <functional> | ||
| #include <string> | ||
| #include <boost/program_options.hpp> | ||
| #include <boost/exception/diagnostic_information.hpp> | ||
| #include <TFile.h> | ||
| #include <TKey.h> | ||
|
|
||
| namespace bpo = boost::program_options; | ||
| using namespace o2::quality_control::core; | ||
| using namespace o2::quality_control::repository; | ||
|
|
||
| int main(int argc, const char* argv[]) | ||
| { | ||
| size_t objectsUploaded = 0; | ||
|
|
||
| try { | ||
| bpo::options_description desc{ "Options" }; | ||
| desc.add_options() // | ||
| ("help,h", "Help screen") // | ||
| ("input-file", bpo::value<std::string>()->default_value("./QAResults.root"), "Path to the ROOT file with objects to insert.") // | ||
| ("qcdb-url", bpo::value<std::string>()->default_value("ccdb-test.cern.ch:8080"), "URL to the QCDB.") // | ||
| ("task-name", bpo::value<std::string>(), "Name of the task to which the objects belong. Use / to make directories") // | ||
| ("detector-code", bpo::value<std::string>()->default_value("TST"), "3-letter detector code. Put AOD for analysis tasks") // | ||
| ("validity-start", bpo::value<uint64_t>()->default_value(0), "Start of objects validity in ms since epoch") // | ||
| ("validity-end", bpo::value<uint64_t>()->default_value(0), "End of objects validity in ms since epoch") // | ||
| ("run-number", bpo::value<uint64_t>(), "Run number of objects (put 0 for many runs)") // | ||
| ("period-name", bpo::value<std::string>()->default_value("unknown"), "Period name of the objects") // todo one could ask logbook | ||
| ("pass-name", bpo::value<std::string>()->default_value("unknown"), "Calib/reco/sim pass name") // | ||
| ("provenance", bpo::value<std::string>()->default_value("qc"), "Object path prefix used to mark if data comes from detector (use qc) or simulation (use qc_mc)"); | ||
|
|
||
| bpo::variables_map vm; | ||
| store(parse_command_line(argc, argv, desc), vm); | ||
| notify(vm); | ||
|
|
||
| if (vm.count("help")) { | ||
| // no infologger here, because the message is too long. | ||
| std::cout << desc << std::endl; | ||
| return 0; | ||
| } | ||
|
|
||
| /// Read and validate arguments | ||
| auto inputFilePath = vm["input-file"].as<std::string>(); | ||
| auto qcdbUrl = vm["qcdb-url"].as<std::string>(); | ||
| auto taskName = vm["task-name"].as<std::string>(); | ||
| auto detectorCode = vm["detector-code"].as<std::string>(); | ||
| auto validityStart = vm["validity-start"].as<uint64_t>(); | ||
| auto validityEnd = vm["validity-end"].as<uint64_t>(); | ||
| auto runNumber = vm["run-number"].as<uint64_t>(); | ||
| auto periodName = vm["period-name"].as<std::string>(); | ||
| auto passName = vm["pass-name"].as<std::string>(); | ||
| auto provenance = vm["provenance"].as<std::string>(); | ||
|
|
||
| if (validityStart == 0) { | ||
| validityStart = CcdbDatabase::getCurrentTimestamp(); | ||
| } | ||
| if (validityEnd == 0) { | ||
| validityEnd = validityStart + 1000ull * 60 * 60 * 24 * 365 * 10; | ||
| } | ||
| if (validityStart > validityEnd) { | ||
| throw std::runtime_error("Validity start (" + std::to_string(validityStart) + ") is further in the future than validity end (" + std::to_string(validityEnd) + ")"); | ||
| } | ||
| if (provenance != "qc" && provenance != "qc_mc") { | ||
| throw std::runtime_error("Provenance should be either 'qc' (for data from detector) or 'qc_mc' (for simulated data), while '" + provenance + "' was given."); | ||
| } | ||
|
|
||
| /// Open ROOT file | ||
| auto* file = new TFile(inputFilePath.c_str(), "READ"); | ||
| if (file->IsZombie()) { | ||
| throw std::runtime_error("File '" + inputFilePath + "' is zombie."); | ||
| } | ||
| if (!file->IsOpen()) { | ||
| throw std::runtime_error("Failed to open the file: " + inputFilePath); | ||
| } | ||
| ILOG(Info) << "Input file '" << inputFilePath << "' successfully open." << ENDM; | ||
|
|
||
| /// Open CCDB interface | ||
| CcdbDatabase database; | ||
| database.connect(qcdbUrl, "", "", ""); | ||
|
|
||
| /// Upload the objects | ||
| std::function<void(TDirectoryFile*)> browseFileAndUpload = [&](TDirectoryFile* directory) { | ||
| TIter next(directory->GetListOfKeys()); | ||
| TKey* key; | ||
| while ((key = (TKey*)next())) { | ||
| auto storedTObj = directory->Get(key->GetName()); | ||
| if (storedTObj != nullptr) { | ||
| if (storedTObj->InheritsFrom("TDirectoryFile")) { | ||
| browseFileAndUpload(dynamic_cast<TDirectoryFile*>(storedTObj)); | ||
| } else { | ||
| std::shared_ptr<MonitorObject> mo = std::make_shared<MonitorObject>(storedTObj, taskName, "unknown", detectorCode, runNumber, periodName, passName, provenance); | ||
| mo->setIsOwner(false); | ||
| database.storeMO(mo, validityStart, validityEnd); | ||
| objectsUploaded++; | ||
| } | ||
| } | ||
| delete storedTObj; | ||
| } | ||
| }; | ||
|
|
||
| browseFileAndUpload(file); | ||
|
|
||
| file->Close(); | ||
| delete file; | ||
|
|
||
| database.disconnect(); | ||
|
|
||
| } catch (const bpo::error& ex) { | ||
| ILOG(Error, Ops) << "Exception caught: " << ex.what() << ENDM; | ||
| return 1; | ||
| } catch (const boost::exception& ex) { | ||
| ILOG(Error, Ops) << "Exception caught: " << boost::current_exception_diagnostic_information(true) << ENDM; | ||
| return 1; | ||
| } | ||
|
|
||
| if (objectsUploaded > 0) { | ||
| ILOG(Info, Support) << "Successfully uploaded " << objectsUploaded << " objects to the QCDB." << ENDM; | ||
| } else { | ||
| ILOG(Info, Support) << "No objects were uploaded to the QCDB. Maybe the file is empty?" << ENDM; | ||
| } | ||
| return 0; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you actually want default values here, in particular 0 ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For validity and filters: I don't think there is a good way to retrieve those values from analysis tasks, so I would not bother people too much about it before we can offer a solution.
For run number: indeed we could require an explicitly stated value, I will fix this. If it is multi-run, people should be aware of it and put 0 then.