Lots of warning messages when running 3ru_3df_multirun_test.py

[bieryAtFnal]

These were seen with the N22-05-19 nightly build and the HEAD of develop in daqconf and the kbiery/integtest_updates_for3.0 branch in dfmodules:

Problem(s) found in logfile /tmp/pytest-of-biery/pytest-638/run0/log_trigger_3334.txt:
2022-May-19 11:01:04,454 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 2 element 0. Set start time 79554162188040000 but last sent time 79554162191290000 DAQModule: tazipper

2022-May-19 11:01:06,960 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 2 element 0. Set start time 79554162188050000 but last sent time 79554162195140000 DAQModule: tazipper

2022-May-19 11:01:13,940 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=1 Oldest stored TS=79554162196262608 Start of window TS=79554162089472543 End of window TS=79554162089472893 Estimated newest stored TS=79554162089472543 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,001 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=2 Oldest stored TS=79554162196262608 Start of window TS=79554162089784418 End of window TS=79554162089784768 Estimated newest stored TS=79554162089784418 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,062 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=3 Oldest stored TS=79554162196262608 Start of window TS=79554162090098793 End of window TS=79554162090099093 Estimated newest stored TS=79554162090098793 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,124 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=4 Oldest stored TS=79554162196262608 Start of window TS=79554162090390868 End of window TS=79554162090391243 Estimated newest stored TS=79554162090390868 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,155 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=5 Oldest stored TS=79554162196262608 Start of window TS=79554162090838958 End of window TS=79554162090840958 Estimated newest stored TS=79554162090838958 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,185 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=6 Oldest stored TS=79554162196262608 Start of window TS=79554162090702468 End of window TS=79554162090702818 Estimated newest stored TS=79554162090702468 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,247 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=7 Oldest stored TS=79554162196262608 Start of window TS=79554162091016868 End of window TS=79554162091017218 Estimated newest stored TS=79554162091016868 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,308 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=8 Oldest stored TS=79554162196262608 Start of window TS=79554162091309218 End of window TS=79554162091309568 Estimated newest stored TS=79554162091309218 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,369 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=9 Oldest stored TS=79554162196262608 Start of window TS=79554162091620618 End of window TS=79554162091620968 Estimated newest stored TS=79554162091620618 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,431 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=10 Oldest stored TS=79554162196262608 Start of window TS=79554162091934943 End of window TS=79554162091935243 Estimated newest stored TS=79554162091934943 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,482 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=11 Oldest stored TS=79554162196262608 Start of window TS=79554162092334298 End of window TS=79554162092336298 Estimated newest stored TS=79554162092334298 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,492 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=12 Oldest stored TS=79554162196262608 Start of window TS=79554162092231518 End of window TS=79554162092231618 Estimated newest stored TS=79554162092231518 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,553 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=13 Oldest stored TS=79554162196262608 Start of window TS=79554162092538793 End of window TS=79554162092539243 Estimated newest stored TS=79554162092538793 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,614 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=14 Oldest stored TS=79554162196262608 Start of window TS=79554162092853143 End of window TS=79554162092853618 Estimated newest stored TS=79554162092853143 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,676 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=15 Oldest stored TS=79554162196262608 Start of window TS=79554162093154218 End of window TS=79554162093154343 Estimated newest stored TS=79554162093154218 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,737 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=16 Oldest stored TS=79554162196262608 Start of window TS=79554162093456818 End of window TS=79554162093457193 Estimated newest stored TS=79554162093456818 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,798 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=17 Oldest stored TS=79554162196262608 Start of window TS=79554162093771643 End of window TS=79554162093772093 Estimated newest stored TS=79554162093771643 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,819 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=18 Oldest stored TS=79554162196262608 Start of window TS=79554162094140028 End of window TS=79554162094142028 Estimated newest stored TS=79554162094140028 Requestor=fragments_to_dataflow0

2022-May-19 11:01:14,860 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=19 Oldest stored TS=79554162196262608 Start of window TS=79554162094073868 End of window TS=79554162094073968 Estimated newest stored TS=79554162094073868 Requestor=fragments_to_dataflow1

2022-May-19 11:01:14,921 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=20 Oldest stored TS=79554162196262608 Start of window TS=79554162094374943 End of window TS=79554162094375268 Estimated newest stored TS=79554162094374943 Requestor=fragments_to_dataflow2

2022-May-19 11:01:14,982 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=21 Oldest stored TS=79554162196262608 Start of window TS=79554162094690068 End of window TS=79554162094690393 Estimated newest stored TS=79554162094690068 Requestor=fragments_to_dataflow0

2022-May-19 11:01:15,054 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=22 Oldest stored TS=79554162196262608 Start of window TS=79554162094995943 End of window TS=79554162094996118 Estimated newest stored TS=79554162094995943 Requestor=fragments_to_dataflow1

2022-May-19 11:01:15,105 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=23 Oldest stored TS=79554162196262608 Start of window TS=79554162095292993 End of window TS=79554162095293493 Estimated newest stored TS=79554162095292993 Requestor=fragments_to_dataflow2

2022-May-19 11:01:15,147 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=24 Oldest stored TS=79554162196262608 Start of window TS=79554162095822613 End of window TS=79554162095824613 Estimated newest stored TS=79554162095822613 Requestor=fragments_to_dataflow0

2022-May-19 11:01:15,167 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=25 Oldest stored TS=79554162196262608 Start of window TS=79554162095608218 End of window TS=79554162095608568 Estimated newest stored TS=79554162095608218 Requestor=fragments_to_dataflow1

2022-May-19 11:01:15,239 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=26 Oldest stored TS=79554162196262608 Start of window TS=79554162095914893 End of window TS=79554162095915343 Estimated newest stored TS=79554162095914893 Requestor=fragments_to_dataflow2

2022-May-19 11:01:15,290 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=27 Oldest stored TS=79554162196262608 Start of window TS=79554162096211168 End of window TS=79554162096211418 Estimated newest stored TS=79554162096211168 Requestor=fragments_to_dataflow0

2022-May-19 11:01:15,361 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=28 Oldest stored TS=79554162196262608 Start of window TS=79554162096526393 End of window TS=79554162096526718 Estimated newest stored TS=79554162096526393 Requestor=fragments_to_dataflow1

2022-May-19 11:01:15,423 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=29 Oldest stored TS=79554162196262608 Start of window TS=79554162096832993 End of window TS=79554162096833468 Estimated newest stored TS=79554162096832993 Requestor=fragments_to_dataflow2

2022-May-19 11:01:15,474 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=30 Oldest stored TS=79554162196262608 Start of window TS=79554162097129243 End of window TS=79554162097129543 Estimated newest stored TS=79554162097129243 Requestor=fragments_to_dataflow0

2022-May-19 11:01:16,028 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=41 Oldest stored TS=79554162196262608 Start of window TS=79554162099883518 End of window TS=79554162099883793 Estimated newest stored TS=79554162099883518 Requestor=fragments_to_dataflow2 -- 10 similar messages suppressed, last occurrence was at 2022-May-19 11:01:15,976627

2022-May-19 11:01:21,249 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=142 Oldest stored TS=79554162196262608 Start of window TS=79554162125908943 End of window TS=79554162125909268 Estimated newest stored TS=79554162125908943 Requestor=fragments_to_dataflow1 -- 100 similar messages suppressed, last occurrence was at 2022-May-19 11:01:21,199058

2022-May-19 11:01:40,048 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 2 element 0. Set start time 79554162202580000 but last sent time 79554162209010000 DAQModule: tazipper

2022-May-19 11:01:52,055 WARNING [dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::data_request(dunedaq::dfmessages::DataRequest, bool) [with ReadoutType = dunedaq::trigger::TCBuffer::TCWrapper; LatencyBufferType = dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper; dunedaq::readoutlibs::DefaultRequestHandlerModel<ReadoutType, LatencyBufferType>::RequestResult = dunedaq::readoutlibs::RequestHandlerConcept<dunedaq::trigger::TCBuffer::TCWrapper, dunedaq::readoutlibs::BinarySearchQueueModeldunedaq::trigger::TCBuffer::TCWrapper >::RequestResult] at /cvmfs/dunedaq-development.opensciencegrid.org/spack-nightly/N22-05-19/spack-0.17.1/opt/spack/gcc-8.2.0/readoutlibs-N22-05-19-hsel77poocyyzvrmeo4efgzgosgqjbym/include/readoutlibs/models/DefaultRequestHandlerModel.hpp:647] GeoID[type: DataSelection, region: 20000, element: 0] Trigger Matching result with empty fragment: TS match result on link 0: Trigger number=171 Oldest stored TS=79554162210619333 Start of window TS=79554162114070728 End of window TS=79554162114072728 Estimated newest stored TS=79554162113716218 Requestor=fragments_to_dataflow0 -- 417 similar messages suppressed, last occurrence was at 2022-May-19 11:01:51,994391

2022-May-19 11:02:08,099 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 2 element 0. Set start time 79554162188680000 but last sent time 79554162191850000 DAQModule: tazipper

2022-May-19 11:02:10,518 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 1 element 0. Set start time 79554162191890000 but last sent time 79554162195320000 DAQModule: tazipper

2022-May-19 11:02:10,603 WARNING [bool dunedaq::trigger::TriggerZipper::proc_one() [with TSET = dunedaq::trigger::Settriggeralgs::TriggerActivity] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-19-7nt5bqbowokrm3kzeih27tcfiunykqed/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 2 element 0. Set start time 79554162188700000 but last sent time 79554162195320000 DAQModule: tazipper

[philiprodrigues]

I think the "Trigger Matching result with empty fragment" errors are coming from TCs arriving not in time order, and should be fixed by the two PRs:

DUNE-DAQ/readoutlibs#24
#126

I'm still working on the "tardy input set" errors

[bieryAtFnal]

With the N22-05-23 Spack nightly, I'm still seeing the following warning messages:

22-May-23 08:50:25,337 WARNING [bool dunedaq::trigger::TriggerZipper<TSET>::proc_one() [with TSET = dunedaq::trigger::Set<triggeralgs::TriggerActivity>] at /tmp/root/spack-stage/spack-stage-trigger-N22-05-23-jzcm4ucluzztjfwbdouocthr4jnbezbx/spack-src/plugins/TriggerZipper.hpp:217] Tardy input set from region 1 element 0. Set start time 79554162208950000 but last sent time 79554162213720000 DAQModule: tazipper

[philiprodrigues]

Do you see a lot of those messages, or just a few?

[bieryAtFnal]

Lots.

[philiprodrigues]

I was not able to fix this in time for 3.0, so we've worked around it in the dfmodules integration tests: DUNE-DAQ/dfmodules#189

[philiprodrigues]

I think these are fixed by the combination of #133 and DUNE-DAQ/daqconf#101

[philiprodrigues]

The changes in trigger and daqconf that fix this have been merged, so closing.