Added DaqSource emulator using DaqFakeReader #12985

jfernan2 · 2016-01-19T10:09:06Z

Ported all plugins from the old (CMSSW_6_X_Y or older) to recent CMSSW versions making use of DaqFakeReader

cmsbuild · 2016-01-19T10:09:26Z

A new Pull Request was created by @jfernan2 for CMSSW_8_0_X.

It involves the following packages:

IORawData/DTCommissioning

@diguida, @cerminar, @cmsbuild, @franzoni, @mmusich, @davidlange6 can you please review it and eventually sign? Thanks.
@tocheng this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

Following commands in first line of a comment are recognized

+1|approve[d]|sign[ed]: L1/L2's to approve it
-1|reject[ed]: L1/L2's to reject it
assign <category>[,<category>[,...]]: L1/L2's to request signatures from other categories
unassign <category>[,<category>[,...]]: L1/L2's to remove signatures from other categories
hold: L1/all L2's/release manager to mark it as on hold
unhold: L1/user who put this PR on hold
merge: L1/release managers to merge this request
[@cmsbuild,] please test: L1/L2 and selected users to start jenkins tests
[@cmsbuild,] please test with cms-sw/cmsdist#<PR>: L1/L2 and selected users to start jenkins tests using externals from cmsdist

mmusich · 2016-01-26T14:50:02Z

please test

cmsbuild · 2016-01-26T14:50:21Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10743/console

cmsbuild · 2016-01-26T17:00:45Z

-1

Tested at: b1998c4
I found errors in the following addon tests:

cmsDriver.py RelVal -s L1REPACK:GT2 --data --scenario=HeavyIons -n 10 --conditions auto:run2_hlt_HIon --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_HI --magField 38T_PostLS1 --fileout file:RelVal_Raw_HIon_DATA.root --filein /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root : FAILED - time: date Tue Jan 26 17:14:40 2016-date Tue Jan 26 17:13:32 2016 s - exit: 23552
cmsRun /afs/cern.ch/cms/sw/ReleaseCandidates/vol1/slc6_amd64_gcc493/cms/cmssw-patch/CMSSW_8_0_X_2016-01-25-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Tue Jan 26 17:14:40 2016-date Tue Jan 26 17:13:32 2016 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Tue Jan 26 17:14:40 2016-date Tue Jan 26 17:13:32 2016 s - exit: 21504

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12985/10743/summary.html

jfernan2 · 2016-01-26T17:34:23Z

Hi,

I cannot access the summary log to find out the reason of the error, I get

Authorization Failed

Based on the information provided to this application about you, you are not authorized to access the resource at "https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12985/10743/summary.html"

Please contact the administrator of this service or application if you believe this to be an error at somebody@cern.ch

when trying to access:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12985/10743/summary.html

jfernan2 · 2016-01-26T17:36:11Z

Is there any way to see the log of the error? The cmsDriver command seems to not be loading anything related to the code pushed in this PR...
Thanks in advance

mmusich · 2016-01-26T17:54:30Z

@jfernan2 that's the stack:

# Conditions read from  CMS_CONDITIONS  via FrontierProd 
26-Jan-2016 17:13:44 CET  Initiating request to open file root://eoscms.cern.ch//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root
[2016-01-26 17:13:45.549158 +0100][Error  ][XRootD            ] [eoscms.cern.ch:1094] Handling error while processing kXR_open (file: /eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response.
%MSG-w XrdAdaptorInternal:  file_open 26-Jan-2016 17:13:45 CET pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root.
%MSG
[2016-01-26 17:13:45.552590 +0100][Error  ][XRootD            ] [eoscms.cern.ch:1094] Handling error while processing kXR_open (file: /eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response.
%MSG-w XrdAdaptorInternal:  file_open 26-Jan-2016 17:13:45 CET pre-events
Failed to open file at URL root://eoscms.cern.ch:1094//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=.
%MSG
26-Jan-2016 17:13:45 CET  Fallback request to file root://xrootd-cms.infn.it//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root
[2016-01-26 17:13:52.345977 +0100][Error  ][XRootDTransport   ] [xrootd.cmsaf.mit.edu:1094 #0.0] No protocols left to try
[2016-01-26 17:13:52.360904 +0100][Error  ][AsyncSock         ] [xrootd.cmsaf.mit.edu:1094 #0.0] Socket error while handshaking: [FATAL] Auth failed
[2016-01-26 17:13:52.361416 +0100][Error  ][PostMaster        ] [xrootd.cmsaf.mit.edu:1094 #0] Unable to recover: [FATAL] Auth failed.
[2016-01-26 17:13:52.361499 +0100][Error  ][XRootD            ] [xrootd.cmsaf.mit.edu:1094] Impossible to send message kXR_open (file: /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1213llrxrd-redir.in2p3.fr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ). Trying to recover.
[2016-01-26 17:13:52.361523 +0100][Error  ][XRootD            ] [xrootd.cmsaf.mit.edu:1094] Handling error while processing kXR_open (file: /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1213llrxrd-redir.in2p3.fr, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Auth failed.
[2016-01-26 17:13:52.524216 +0100][Error  ][XRootD            ] [cms-xrd-transit.cern.ch:1094] Handling error while processing kXR_open (file: /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1098xrdcmsglobal02.cern.ch,+1213cmsxrootd1.fnal.gov1213xrootd.unl.edu,+1213llrxrd-redir.in2p3.fr,xrootd.cmsaf.mit.edu,cmsxrootd1.fnal.gov, mode: 0460, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response.
%MSG-w XrdAdaptorInternal:  file_open 26-Jan-2016 17:13:52 CET pre-events
Failed to open file at URL root://cms-xrd-transit.cern.ch:1094//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1098xrdcmsglobal02.cern.ch,+1213cmsxrootd1.fnal.gov1213xrootd.unl.edu,+1213llrxrd-redir.in2p3.fr,xrootd.cmsaf.mit.edu,cmsxrootd1.fnal.gov.
%MSG
[2016-01-26 17:13:52.657049 +0100][Error  ][XRootD            ] [xrootd.cmsaf.mit.edu:1094] Handling error while processing kXR_open (file: /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1213llrxrd-redir.in2p3.fr,, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [FATAL] Auth failed.
[2016-01-26 17:13:53.102265 +0100][Error  ][XRootD            ] [cms-xrd-transit.cern.ch:1094] Handling error while processing kXR_open (file: /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1098xrdcmsglobal02.cern.ch,+1213cmsxrootd1.fnal.gov1213xrootd.unl.edu,+1213llrxrd-redir.in2p3.fr,,xrootd.cmsaf.mit.edu,cmsxrootd1.fnal.gov, mode: 0660, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response.
%MSG-w XrdAdaptorInternal:  file_open 26-Jan-2016 17:13:53 CET pre-events
Failed to open file at URL root://cms-xrd-transit.cern.ch:1094//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1098xrdcmsglobal02.cern.ch,+1213cmsxrootd1.fnal.gov1213xrootd.unl.edu,+1213llrxrd-redir.in2p3.fr,,xrootd.cmsaf.mit.edu,cmsxrootd1.fnal.gov.
%MSG
----- Begin Fatal Exception 26-Jan-2016 17:13:53 CET-----------------------
An exception of category 'FallbackFileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootFileSequenceBase::initTheFile()
   [3] Calling StorageFactory::open()
   [4] Calling XrdFile::open()
Exception Message:
Failed to open the file 'root://xrootd-cms.infn.it//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] Unable to open file /eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root; No such file or directory
' (errno=3011, code=400). No additional data servers were found.
      [b] Last URL tried: root://eoscms.cern.ch:1094//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=
      [c] Problematic data server: eoscms.cern.ch:1094
      [d] Disabled source: eoscms.cern.ch:1094
      [e] Input file root://eoscms.cern.ch//eos/cms/store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root could not be opened.
Fallback Input file root://xrootd-cms.infn.it//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root also could not be opened.
Original exception info is above; fallback exception info is below.
      [f] XrdCl::File::Open(name='root://xrootd-cms.infn.it//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3011] No servers are available to read the file.
' (errno=3011, code=400). No additional data servers were found.
      [g] Last URL tried: root://cms-xrd-transit.cern.ch:1094//store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root?tried=+1098xrdcmsglobal02.cern.ch,+1213cmsxrootd1.fnal.gov1213xrootd.unl.edu,+1213llrxrd-redir.in2p3.fr,,xrootd.cmsaf.mit.edu,cmsxrootd1.fnal.gov
      [h] Problematic data server: cms-xrd-transit.cern.ch:1094
      [i] Disabled source: cms-xrd-transit.cern.ch:1094
----- End Fatal Exception -------------------------------------------------

jfernan2 · 2016-01-26T17:57:13Z

But, if I'm not mistaken, that's an AAA xrootd error, nothing related to the code in the PR, the input file is not even open...
Thanks

davidlt · 2016-01-26T17:58:01Z

Block was deleted from the sites, a request is made to get them back.

cmsbuild · 2016-01-26T18:03:57Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12985/10743/summary.html

mmusich · 2016-01-27T09:05:57Z

please test

cmsbuild · 2016-01-27T09:06:13Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10768/console
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/11219/console

cmsbuild · 2016-02-19T11:28:20Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12985/11379/summary.html

mmusich · 2016-02-24T20:07:45Z

+1

cmsbuild · 2016-02-24T20:08:05Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

jfernan2 · 2016-03-02T09:08:59Z

Any news on this PR? Thanks in advance

jfernan2 · 2016-03-17T10:30:04Z

Sorry for being tedius but, any news on this PR? Thanks in advance

Added DaqSource emulator using DaqFakeReader

davidlt · 2017-02-11T09:03:29Z

@jfernan2 could you move away from RFIO interface? RFIO was deprecated long time ago and they are decommissioning it. They have disabled write capabilities already. It's not a recommended protocol to read data from CASTOR. They do recommend "xroot" protocol ("root://").

davidlt · 2017-02-11T11:32:08Z

Also don't use XrdClient from xrootd, that's deprecated for many years and they want to remove it in soon to be released xrootd version.

jfernan2 · 2017-02-11T16:09:46Z

@davidlt Sure but this PR was merged almost one year ago... do you want another 80X PR or what do you mean exactly? Thanks

davidlt · 2017-02-11T16:20:23Z

@jfernan2 for the current development release (i.e. CMSSW_9_0_X).

davidlt · 2017-02-11T16:25:41Z

Looks like the new client API in xrood it called XrdCl: https://indico.cern.ch/event/330212/contributions/1718787/attachments/642385/883835/20150128_new_client_ins_outs.pdf

jfernan2 · 2017-03-23T08:17:57Z

@davidlt I have not taken actions on this yet since I do not know how to use the client API. After a request for support to the CERN IT through the Service Portal, I got no answer yet.
I do not find in XrdClFile some functionalities existing in XrdPosix, like XrdPosix_Fseek and XrdPosix_Ftell, which are needed in the current implementation. Do you have by chance some working example with the new API? Thanks in advance

davidlt · 2017-05-10T09:29:46Z

I do known more than a Google search would know.

CC @smuzaffar

davidlt · 2017-05-10T09:32:04Z

I am looking at XrdCl::File doxygen documentation and I see:

XRootDStatus 	Read (uint64_t offset, uint32_t size, void *buffer, ResponseHandler *handler, uint16_t timeout=0)
 
XRootDStatus 	Read (uint64_t offset, uint32_t size, void *buffer, uint32_t &bytesRead, uint16_t timeout=0)
 
XRootDStatus 	Write (uint64_t offset, uint32_t size, const void *buffer, ResponseHandler *handler, uint16_t timeout=0)

XRootDStatus 	Write (uint64_t offset, uint32_t size, const void *buffer, uint16_t timeout=0)

As you can see we have offset arguments for write and read operations. Is that what why you need a seek operation?

davidlt · 2017-05-10T09:35:54Z

Looking at your implementation, I think, your RawFile class would need to store offset (as private member) and use it for write and read operations. The new xrootd API does not not seem to track internally the offset, it's user responsibility (at least it looks like that by a very brief look).

davidlt · 2017-05-10T09:39:59Z

I also see that you already have xrootdFlag and rfioFlag. Thus basically this would be removing rfioFlag and making xrootdFlag permanent.

jfernan2 · 2017-05-10T10:49:07Z

@davidlt thanks!
However, removing rfio part is easy and it was in my implemented changes but not commited yet.
My worry was for your request to move to the new XrdCl since I see no documentation at all apart from doxygen. The seek operation was using offset yes, but from the current XrdPosix paradigm to the new XRootDStatus I don't know how to proceed without a working example.

davidlt · 2017-05-10T11:07:00Z

It's enough to get rid of rfio at this point. Btw, why in the first place you added both (xrootd and rfio) and if anyone used rfio?

jfernan2 · 2017-05-10T11:11:53Z

I just inherited the code from another developer and moved from DT private code to CMSSW releases. I guess the option was there for the time period when castor lived together with eos... now it is clearly obsolete, but it was used to read local and miniDAQ files.
I will create a new PR in 91X to remove rfio option. I'll keep you in the loop if you don't mind. Thank you very much

davidlt · 2017-05-10T11:16:34Z

92X (master branch of CMSSW) is fine. 91X is already a closed release IIUC. 92X was opened a few days ago.

Thanks.

davidlt · 2017-05-10T11:18:46Z

Btw, you can test your changes with SCRAM_ARCH set to slc6_amd64_gcc700.

export SCRAM_ARCH=slc6_amd64_gcc700
scram p CMSSW_9_2_X_2017-05-09-2300
cd CMSSW_9_2_X_2017-05-09-2300
cmsenv
git cms-addpkg IORawData/DTCommissioning
# or
git cms-merge-topic <github_account>:<branch>
# if you already have a working branch

Here is the latest from integration builds: https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc700/CMSSW_9_2_X_2017-05-09-2300/IORawData/DTCommissioning

Added DaqSource emulator using DaqFakeReader

b1998c4

cmsbuild added this to the Next CMSSW_8_0_X milestone Jan 19, 2016

cmsbuild added alca-pending pending-signatures tests-pending orp-pending comparison-pending labels Jan 19, 2016

cmsbuild added tests-started and removed tests-pending labels Jan 26, 2016

cmsbuild added tests-rejected and removed tests-started labels Jan 26, 2016

cmsbuild added comparison-available and removed comparison-pending labels Jan 26, 2016

cmsbuild added tests-pending comparison-pending and removed comparison-available tests-rejected tests-pending labels Jan 27, 2016

cmsbuild added tests-approved and removed tests-started labels Feb 19, 2016

cmsbuild added comparison-available and removed comparison-pending labels Feb 19, 2016

cmsbuild added alca-approved fully-signed and removed alca-pending pending-signatures labels Feb 24, 2016

davidlange6 added a commit that referenced this pull request Mar 22, 2016

Merge pull request #12985 from jfernan2/added-DaqSource-emulator

55097c5

Added DaqSource emulator using DaqFakeReader

davidlange6 merged commit 55097c5 into cms-sw:CMSSW_8_0_X Mar 22, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added DaqSource emulator using DaqFakeReader #12985

Added DaqSource emulator using DaqFakeReader #12985

jfernan2 commented Jan 19, 2016

cmsbuild commented Jan 19, 2016

mmusich commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

mmusich commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

davidlt commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

mmusich commented Jan 27, 2016

cmsbuild commented Jan 27, 2016

cmsbuild commented Feb 19, 2016

mmusich commented Feb 24, 2016

cmsbuild commented Feb 24, 2016

jfernan2 commented Mar 2, 2016

jfernan2 commented Mar 17, 2016

davidlt commented Feb 11, 2017

davidlt commented Feb 11, 2017

jfernan2 commented Feb 11, 2017

davidlt commented Feb 11, 2017

davidlt commented Feb 11, 2017

jfernan2 commented Mar 23, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

jfernan2 commented May 10, 2017

davidlt commented May 10, 2017

jfernan2 commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

Added DaqSource emulator using DaqFakeReader #12985

Added DaqSource emulator using DaqFakeReader #12985

Conversation

jfernan2 commented Jan 19, 2016

cmsbuild commented Jan 19, 2016

mmusich commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

mmusich commented Jan 26, 2016

jfernan2 commented Jan 26, 2016

davidlt commented Jan 26, 2016

cmsbuild commented Jan 26, 2016

mmusich commented Jan 27, 2016

cmsbuild commented Jan 27, 2016

cmsbuild commented Feb 19, 2016

mmusich commented Feb 24, 2016

cmsbuild commented Feb 24, 2016

jfernan2 commented Mar 2, 2016

jfernan2 commented Mar 17, 2016

davidlt commented Feb 11, 2017

davidlt commented Feb 11, 2017

jfernan2 commented Feb 11, 2017

davidlt commented Feb 11, 2017

davidlt commented Feb 11, 2017

jfernan2 commented Mar 23, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017

jfernan2 commented May 10, 2017

davidlt commented May 10, 2017

jfernan2 commented May 10, 2017

davidlt commented May 10, 2017

davidlt commented May 10, 2017