-
Notifications
You must be signed in to change notification settings - Fork 0
/
local.yaml
1110 lines (1103 loc) · 55.8 KB
/
local.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
A2330:
description: Date for Authentication and Authorization hardware to ship from NCSA
to Chile
level: 4
A4369.9:
description: Date for Base Data Center hardware to ship from NCSA to Chile
level: 4
A4640:
description: Date for ComCam support systems to ship from NCSA to Tucson
level: 4
A4670:
description: Start of travel to Chile to install hardware and systems at the Base
Data Center
level: 4
A4680:
description: End of travel to Chile to install hardware and systems at the Base
Data Center
level: 4
A4750:
description: Authorization and Authentication systems installed at the Summit.
level: 4
A4860:
description: ComCam Test Stand ready for access by NCSA staff
level: 4
A5090:
description: Arrival at NCSA of hardware to support ComCam testing and integration
level: 4
CAMM6995:
comment: ''
description: ''
test_spec: ''
DLP-33:
description: Spectrum link operational at 100 Gbps from Santiago - Boca Raton, including
terrestrial segment from Santiago - Sao Paolo, and Monet submarine cable from
Sao Paolo - Boca Raton
DLP-46:
description: Both primary (100 Gbps) and secondary (40 - 100 Gbps) circuits from
La Serena - Santiago are operational and tested with loads to full capacity.
DLP-464:
description: Installed the Base LAN equipment in the Base Data Center and Base Office
Building technical closets, and cabling connecting them for the Base Campus and
Control Networks. Tested with data volumes and access points equivalent to operational
staffing and control servers for commissioning camera observing.
DLP-465:
description: Installed the Base LAN equipment in the Base Data Center and Base Office
Building technical closets, and cabling connecting them for the Base Campus and
Control Networks. Tested with data volumes and access points equivalent to operational
staffing and control servers for full camera observing.
DLP-466:
description: Installed the Base LAN equipment in the Chilean DAC. Tested with data
volumes and access points equivalent to anticipated DAC user workload contemporaneous
with ATS observing.
DLP-467:
description: Installed the Base LAN equipment in the Chilean DAC. Tested with data
volumes and access points equivalent to anticipated DAC user workload contemporaneous
with commissioning camera observing.
DLP-468:
description: Installed the Base LAN equipment in the Chilean DAC. Tested with data
volumes and access points equivalent to anticipated DAC user workload contemporaneous
with full camera observing.
DLP-526:
description: This milestone represents the availability of an archive center to
host LSST DR1 data. This includes an science platform interface with the implied
services for Data Access, a portal and Notebooks.
DLP-528:
description: This is complete in .. data/pmcs/201904-ME.xls
DLP-533:
description: This milestone represents the availability of the base center in Las
Serena. This includes DM staging machines, the commissioning cluster. It also includes
the Chilean Data Access center machines. The later will only be deployed after
the US DAC is up.
DLP-538:
description: approval of LCR-1494 in June 2019 allows this to be closed, LSE-400
covers the items.
DLP-541:
description: Complete with LCR-1788 June 2019
DLP-552:
description: This pipeline software release will precede LDM-503-12a. This will
have all functionality required to process data from LSSTCam. It will not have
the full functionality for data release processing.
DLP-69:
description: Boca Raton - Atlanta and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse
circuits available and tested at capacity.
DLP-70:
description: Miami - Atlanta, and ESNet Atlanta - Chicago 2 x 20 Gbps path diverse
circuits available and tested at capacity.
DLP-876:
description: Boca Raton - Atlanta and ESNet Atlanta - Chicago 2 x 100 Gbps path
diverse circuits available and tested at capacity.
DLP-877:
description: Miami - Atlanta and ESNet Atlanta - Chicago 2 x 100 Gbps path diverse
circuits available and tested at capacity.
DM-AP-1:
aka:
- SFM1
name: Basic single frame measurement pipeline.
DM-AP-10:
aka:
- SFM2
description: "The Single Frame Processing system is now substantially complete and\
\ provides all functionality described in LDM-151 \xA73.1. It supports all expected\
\ interfaces, data-flows, and products, but the details of the algorithms are\
\ expected to undergo continued refinement."
name: Advanced single frame measurement pipeline for Alert Production.
DM-AP-11:
aka:
- DIFF2
description: "Template generation, including code required to build templates in\
\ both Data Release and Alert Production science payloads, can now product templates\
\ which are corrected for the effects of Differential Chromatic Refraction (DCR),\
\ as described in LDM-151 \xA76.18. Further, noise in image differences can be\
\ \u201Cdecorrelated\u201D, as described in LDM-151 \xA76.19.1. Both of these\
\ represents initial capabilities, providing realistic interfaces, data-flows\
\ and products; ongoing refinement of the algorithms being used is expected."
name: Difference imaging includes noise decorrelation and correction for differential
chromatic refraction.
DM-AP-12:
aka:
- DIFF3
description: "The Alert Generation science payload is now agnostic to the PSF of\
\ the template, as described in LDM-151 \xA73.2.4. This represents an initial\
\ capability, providing realistic interfaces, data-flows and products; ongoing\
\ refinement of the algorithm being used is expected."
name: Difference imaging is now agnostic to the PSF of the template image.
DM-AP-13:
aka:
- DIFFMEAS2
description: "The Alert Generation science payload now provides the capability to\
\ fit a trailed point source model to detections made on difference images, as\
\ described in LDM-151 \xA73.2.4. This represents an initial capability, providing\
\ realistic interfaces, data-flows and products; ongoing refinement of the algorithm\
\ being used is expected."
name: Trailed source measurement on difference images.
DM-AP-14:
aka:
- ALERTDIST2
description: "The Alert Filtering Service (LDM-151 \xA73.3.4; LSE-163 \xA73.5.2)\
\ can now be deployed as part of a prototype Alert Distribution system (LDM-151\
\ \xA73.3). This service is substantially complete, and supports all expected\
\ interfaces, data-flows, and products, but may not yet provide the complete set\
\ of filtering capabilities and/or may not perform at the level required to distribute\
\ the full LSST alert stream."
name: Alert filtering system available.
DM-AP-15:
aka:
- INTEGRATION1
description: "The Alert Distribution system (LDM-151 \xA73.3) is now available for\
\ deployment at the LSST Data Facility. The system is substantially complete,\
\ and supports all expected interfaces, data-flows, and products, but the details\
\ of the algorithms may undergo continued refinement."
name: Alert distribution system fully integrated.
DM-AP-16:
aka:
- INTEGRATION2
description: "The Alert Production system (comprising Single Frame Processing, LDM-151\
\ \xA73.1; Alert Generation, LDM-151 \xA73.3; and Precovery and Forced Photometry,\
\ LDM-151 \xA73.4) is now available for deployment at the LSST Data Facility.\
\ The system is substantially complete, and supports all expected interfaces,\
\ data-flows and products, but the details of the algorithms may undergo continued\
\ refinement."
name: Full integration of the Alert Production system within the operational environment.
DM-AP-17:
aka:
- MOPS
description: "All of the algorithmic components which comprise the Solar System\
\ Processing (SSP) pipeline, as described in LDM-151 \xA7\xA76.25\u20136.28, are\
\ available for integration into science payloads. These components must provide\
\ realistic interfaces, data-flows and products, but it is expected that the details\
\ of the algorithms will undergo continued refinement."
name: Solar System Processing pipeline available.
DM-AP-18:
description: The AP Pipeline is now capable of generating per-DIASource alert packets,
which comply with the current version of the LSST alert schema. This milestone
only covers packet generation; success involves only those serialized alerts being
e.g. written to disk, and does not imply that an alert transport system is available.
DM-AP-2:
aka:
- DIFF1
name: Alard & Lupton-style image differencing.
DM-AP-3:
aka:
- DIFFMEAS1
name: Point source & dipole measurement on difference images.
DM-AP-6:
aka:
- ALERTDIST1
name: Alert format defined & queue system available.
DM-AP-7:
aka:
- ISR1
name: Basic instrument signature removal (ISR) capability.
DM-AP-8:
aka:
- ISR2
name: Advanced ISR, including ghost and linear feature masking, correction for the
Brighter-Fatter effect and compensation for pixel response non-uniformity.
DM-ARCH-PR:
description: After we have Gen3 butler and have completed LDM-503-11 we need to
review the provenance we collect and see it is sufficient for our needs of reproducibility
and diagnosis of problems.
DM-DAX-10:
description: Demonstrate Butler configuration able to gather all data pertaining
to a particular object, such as retrieving cutouts from all warps at a given box
on a skymap.
DM-DAX-12:
description: Generation 3 middleware, consisting of Butler updates and the PipelineTask
framework, are now able to support existing science payloads with no loss of functionality
over previous versions. That is, Gen 3 may not be feature complete, but Gen 2
can be retired with no functionality regressions across the subsystem.
DM-DAX-13:
description: The Generation 2 middleware is deprecated, and may be removed from
the codebase. No current science payload and no current execution service makes
use of it.
DM-DAX-2:
comment: 'PMCS calls this: dbserv V1 API (TAP, w/ async. requests)'
description: An IVOA TAP service supporting both synchronous and asynchronous query
models, integrated with the LSP authentication architecture, and able to access
catalog data hosted in a downstream in Qserv instance.
name: Query service supporting IVOA TAP protocol, w/ support for asynchronous queries
DM-DAX-3:
comment: 'PMCS calls this: imgserv V1 API (SODA)'
description: An IVOA SODA service supporting both synchronous and asynchronous query
models, integrated the the LSP authentication architecture, and able to provide
cutout retrieval service from images hosted in a downstream Butler-fronted archive.
name: Image cutout service supporting IVOA SODA protocol
DM-DAX-4:
comment: 'PMCS calls this: metaserv V1 API (SIAv2)'
description: An IVOA SIAv2 service supporting both synchronous and asynchronous
query models, integrated with the LSP authentication architecture, and able to
provide discovery and retrieval of image metadata within an LSP instance.
name: Metadata service supporting IVOA SIAv2 protocol
DM-DAX-5:
description: Existing Qserv ingest tooling is highly manual and poorly parallelized. Develop
new tooling to address these issues and support ingest campaigns at the scale
needed for DRP.
DM-DAX-7:
description: Given a stood-up data backbone instance, demonstrate instantiation/configuration
of a Butler Gen3 instance capable of retrieving images from that backbone instance. (As
currently conceived, the data backbone is either a GPFS filesystem or object store,
in combination with metadata that is a superset of that maintained in a Butler
Gen3 registry. Depending on particular technologies and implementation strategies
chosen for the data backbone, this may be either a "simple matter of configuration"
of existing Butler connectors, or may require some additional custom connectors
to be written.)
DM-DAX-8:
description: Demonstrate a functional PipelineTask with scatter/gather functionality,
such as Jointcal, SkyCorrect, or FGCM.
DM-DAX-9:
description: ''
DM-DRP-1:
aka:
- DRP-MS-INT-1
name: 'HSC merger complete: all functionality deployed for the most recent HSC data
release processing is now available within the LSST stack.'
DM-DRP-10:
aka:
- DRP-MS-CPP-8
description: "The Calibration Products Production science payload can now generate\
\ an optical ghost model. This is described in LDM-151 \xA74.3.18, but further\
\ refinement of requirements will be required to fully establish the details of\
\ this capability."
name: Calibration products include an optical ghost model.
DM-DRP-11:
aka:
- DRP-MS-OBJCHAR-8
description: The Data Release Production science payload now generates results which
are compliant with the LSST Science Data Model, and which are suitable for database
ingest.
name: Pipelines code provides supports for database ingestion of results.
DM-DRP-12:
aka:
- DRP-MS-IMCHAR-6
name: Background estimation over the full visit.
DM-DRP-13:
aka:
- DRP-MS-IMCHAR-2
description: "The Data Release Production science payload now includes code for\
\ estimating the PSF over the full focal plane, as described in LDM-151 \xA76.11.3.\
\ This does not require that the PSF model be \u201Cphysically motivated\u201D\
\ (ie, decomposed into atmospheric and optical components). Further, this represents\
\ an initial capability, providing realistic interfaces, data-flows and products;\
\ ongoing refinement of the algorithm being used is expected."
name: PSF estimation over the full visit.
DM-DRP-14:
aka:
- DRP-MS-OBJCHAR-9
name: Insertion of simulated sources into the data stream to check pipeline performance.
DM-DRP-15:
aka:
- DRP-MS-COADD-3
name: All varieties of coadd required for object detection and characterization
are now produced during normal pipeline operation (although not necessarily at
the ultimately required level of fidelity).
DM-DRP-16:
aka:
- DRP-MS-IMCHAR-1
name: Global photometric fitting (e.g. Burke et al. Forward Global Calibration Method).
DM-DRP-17:
aka:
- DRP-MS-IMCHAR-4
name: Simultaneous photometric and astrometric fitting to multiple exposures.
DM-DRP-18:
aka:
- DRP-MS-DET-1
name: Initial multi-band deblending algorithm available.
DM-DRP-19:
aka:
- DRP-MS-QA-2
name: QA metrics are generated during pipeline execution.
DM-DRP-2:
aka:
- DRP-MS-QA-1
name: Basic visualization and quality assessment tools operational on HSC-scale
data volumes.
DM-DRP-20:
aka:
- DRP-MS-CPP-7
description: The Calibration Products Production science payload is now capable
of generating all calibration products which are required to enter commissioning.
name: Refined set of LSST calibration products.
DM-DRP-21:
aka:
- DRP-MS-IMCHAR-5
description: "The Data Release Production science payload now includes an integrated\
\ image characterization pipeline, as described in LDM-151 \xA75.1, including\
\ iterative refinement of the image characterization. This represents an initial\
\ capability, providing realistic interfaces, data-flows and products; ongoing\
\ refinement of the algorithm being used is expected."
name: Integrated image characterization pipeline for Data Release Production.
DM-DRP-22:
aka:
- DRP-MS-COADD-4
description: "The Data Release Production science payload is now capable of generating\
\ templates for use in image differencing, as described in LDM-151 \xA7\xA75.2.3\
\ & 5.2.4. This represents an initial capability, providing realistic interfaces,\
\ data-flows and products; ongoing refinement of the algorithm being used is expected."
name: Template generation integrated with Data Release Production pipelines.
DM-DRP-23:
aka:
- DRP-MS-CPP-10
description: "The Calibration Products Production science payload is now capable\
\ of producing an atmospheric transmission model based on observations from the\
\ Auxiliary Telescope, as described in LDM-151 \xA74.6.1. This milestone does\
\ not presume that the Auxiliary Telescope itself is available for testing: it\
\ is satisfied by the availability of a pipeline that can process appropriate\
\ simulated or precursor data. This represents an initial capability, providing\
\ realistic interfaces, data-flows and products; ongoing refinement of the algorithm\
\ being used is expected."
name: Atmospheric characterization based on data from the Auxiliary Telescope now
available.
DM-DRP-24:
aka:
- DRP-MS-IMCHAR-3
description: "Per LDM-151 \xA76.11.3, the PSF model used by the Data Release Production\
\ science payload is now decomposed into separate contributions from the telescope\
\ optics and the atmosphere."
name: Physically motivated PSF model, including separate characterization of contributions
from the atmosphere and the telescope system.
DM-DRP-25:
aka:
- DRP-MS-OBJCHAR-1
description: "A prototype of the \u201CMultiFit\u201D\_multi-epoch fitting is now\
\ available for testing. This provides the capabilities described in LDM-151 \xA7\
5.5.1, but it is not necessarily integrated with the Data Release Production science\
\ payload (i.e., it may not be run in regular dataset reprocessing, and it may\
\ require separate/additional scripts or ancillary artifacts)."
name: Prototype multi-epoch fitting system available.
DM-DRP-26:
aka:
- DRP-MS-DET-3
description: "The Data Release Production science payload is now capable of finalizing\
\ the definition of Objects which fall within the overlap regions at the edges\
\ of tracts and patches, as described in LDM-151 \xA75.4. This represents an initial\
\ capability, providing realistic interfaces, data-flows and products; ongoing\
\ refinement of the algorithm being used is expected."
name: Overlap resolution at tract & patch boundaries.
DM-DRP-27:
aka:
- DRP-MS-DET-2
description: "The Data Release Production science payload now includes the capability\
\ to build Objects by cross-matching the Source and DIASource tables, as described\
\ in LDM-151 \xA75.3.2. This represents an initial capability, providing realistic\
\ interfaces, data-flows and products; ongoing refinement of the algorithm being\
\ used is expected."
name: 'Object generation: association and assembly of (DIA, coadd, etc) sources
to form objects.'
DM-DRP-28:
aka:
- DRP-MS-COADD-5
description: "The Data Release Production science payload now includes the capability\
\ to generate difference images, and to perform source detection and characterization\
\ on those images, as described in LDM-151 \xA75.2.5. This represents an initial\
\ capability, providing realistic interfaces, data-flows and products; ongoing\
\ refinement of the algorithm being used is expected."
name: Difference images are now a first-class data product during data release processing.
DM-DRP-29:
aka:
- DRP-MS-OBJCHAR-3
description: "The Data Release Production science payload now includes the capability\
\ of fitting moving a moving point source model that includes parallax and proper\
\ motion as free parameters to all objects, as described in LDM-151 \xA76.7.2.\
\ This represents an initial capability, providing realistic interfaces, data-flows\
\ and products; ongoing refinement of the algorithms being used is expected."
name: Moving point source model fitting now available.
DM-DRP-3:
aka:
- DRP-MS-COADD-1
name: PSF-homogenized coadd construction.
DM-DRP-30:
aka:
- DRP-MS-OBJCHAR-2
description: "The Data Release Production science payload now includes the capability\
\ of performing \u201Cforced\u201D photometry at the positions of known Objects\
\ on Processed Visit Images, as described in LDM-151 \xA75.5.2. This represents\
\ an initial capability, providing realistic interfaces, data-flows and products;\
\ ongoing refinement of the algorithm being used is expected."
name: Forced photometry is now performed on individual processed visit images during
data releases.
DM-DRP-31:
aka:
- DRP-MS-OBJCHAR-7
description: "The Data Release Production science payload now includes the capability\
\ of calculating photometric redshifts as part of a data release, as (briefly)\
\ described in LDM-151 \xA75.6.5. This represents an initial capability, providing\
\ realistic interfaces, data-flows and products; ongoing refinement of the algorithm\
\ being used is expected."
name: A photometric redshift is now provided for each object.
DM-DRP-32:
aka:
- DRP-MS-OBJCHAR-6
name: Object classification system available.
DM-DRP-33:
aka:
- DRP-MS-COADD-6
name: Generation of coadded images suitable for use in EPO activities.
DM-DRP-34:
aka:
- DRP-MS-OBJCHAR-5
description: "The Data Release Production science payload now includes the capability\
\ of generating multi-scale maps that describe LSST\u2019s depth and efficiency\
\ at detecting different classes of object as part of a data release, as described\
\ in LDM-151 \xA75.6.3. This represents an initial capability, providing realistic\
\ interfaces, data-flows and products; ongoing refinement of the algorithms being\
\ used is expected."
name: Selection maps are generated during data releases.
DM-DRP-35:
aka:
- DRP-MS-DET-4
description: "The Data Release Production science payload now includes the capability\
\ to perform consistent measurement of all Objects in a patch across multiple\
\ coadds, as described in LDM-151 \xA75.3.4. This represents an initial capability,\
\ providing realistic interfaces, data-flows and products; ongoing refinement\
\ of the algorithms being used is expected."
name: Simultaneous measurement across a suite of coadds representing different bandpasses,
epocs, and flavors.
DM-DRP-36:
aka:
- DRP-MS-OBJCHAR-4
description: "The Data Release Production science payload now generates SSObjects,\
\ by delegating to the Solar System Processing (SSP) pipeline being delivered\
\ by the Alert Production team, as described in LDM-151 \xA75.6.1. This represents\
\ an initial capability, providing realistic interfaces, data-flows and products;\
\ ongoing refinement of the algorithm being used is expected."
name: Solar System Processing integrated to data release processing.
DM-DRP-37:
aka:
- DRP-MS-COADD-2
description: "The Data Release Production science payload now includes code for\
\ generating final estimates of the sky background and artifact masks, as described\
\ in LDM-151 \xA75.2.2. This represents an initial capability, providing realistic\
\ interfaces, data-flows and products; ongoing refinement of the algorithm being\
\ used is expected."
name: Artifact rejection and background matching during coadd construction.
DM-DRP-38:
aka:
- DRP-MS-CPP-2
name: Camera package supporting the Commissioning Camera.
DM-DRP-4:
aka:
- DRP-MS-CPP-6
name: Calibration product generation in support of basic ISR.
DM-DRP-5:
aka:
- DRP-MS-CPP-3
name: Camera package supporting the LSST Camera.
DM-DRP-6:
aka:
- DRP-MS-CPP-1
name: Camera package supporting the Auxiliary Telescope.
DM-DRP-7:
aka:
- DRP-MS-CPP-4
name: Coordinate transformation tool provided for use with the Collimated Beam Projector.
DM-DRP-8:
aka:
- DRP-MS-CPP-5
description: The Calibration Products Production science payload is now capable
of generating those products which are required to calibrate the Auxiliary Telescope.
This represent an initial capability, providing realistic interfaces, data-flows
and products; ongoing refinement of the algorithms being used is expected.
name: Calibration product generation for the Auxiliary Telescope.
DM-DRP-9:
aka:
- DRP-MS-CPP-9
name: Data reduction pipeline for the Auxiliary Telescope.
DM-IT-FIB:
description: Camera fibers ready deployed from the Computer Room to the top of the TMA.
name: Camera Fibers Ready
DM-NCSA-10:
description: The capability to use the existing OCS/CCS to command data acquisition
and hand-off to a test Observatory Operations Data Service (OODS).
DM-NCSA-11:
description: Exercise the handoff between the L1 archiving and data backbone endpoint,
including a monitoring service that tracks files that have been transferred and
received.
DM-NCSA-12:
description: ETL Service in place which can extract, transform and load into the
reformatted EFD DB schema and can ingest EFD Large File Annex into the Data Backbone
at NCSA, recording metadata, provenance and location.
DM-NCSA-13:
description: The Header builder integrated into OCS and demonstrated. Header builder
has identified sources of information for all modes of observing with the Spectrograph.
DM-NCSA-14:
description: Provide a data Backbone endpoint in Chile, allowing distribution over
a wide-area network and ingestion at NCSA.
DM-NCSA-15:
description: Demonstrate the ability to process offline Spectrograph data using
a science pipeline in the Batch Processing Service. Output(s) will be available
in the Data Backbone.
DM-NCSA-16:
description: Process ComCam-scale data through the ISR pipeline to generate QC parameters
used by the observatory.
DM-NCSA-17:
description: QA parameters computed on single-visit processing are converted to
telemetry messages and returned to the OCS system. QA parameters may be generated
in an offline environment through analysis in the Jupyter notebooks.
DM-NCSA-18:
description: Demonstrate the ability to track data in the Data Backbone, query for
data integrity (catalog versus file system), and recover data as needed.
DM-NCSA-19:
description: Capabilities to provide (1) system monitoring to show current system
status; (2) incident response system and workflow in place; (3) process(es) to
address incidents; (4) roles staffed to be able to respond to incidents.
DM-NCSA-20:
description: Implement a Service that acquires raw data from a camera at raft scale
and sends to NCSA for ingestion into the Data Backbone. This is a preliminary
milestone to demonstrate initial capabilities.
DM-NCSA-21:
description: Demonstrated running of processing campaigns in offline mode needed
for the ComCam phase of commissioning to debug the instrument. Likely includes
calibration, ISR, and other pipelines.
DM-NCSA-22:
description: Scientific and operational data products stored in the consolidated
database during ComCam data processing are available for QA assessment of ComCam
data through normal analysis access environments.
DM-NCSA-23:
description: Provide capabilities and processes needed to finalize and prepare a
data release after campaign processing is complete. This includes migration of
data into release schema, tagging of images files, application of permissions
on data, QA work, pushing release products to the DAC, verification of products
pushed to the DAC.
DM-NCSA-24:
description: Provide a Service able to process bulk data according to data release
configurations.
DM-NCSA-25:
description: Provide central job management at NCSA that deploys jobs to CC-IN2P3,
optionally with data transfer or using files that are already located in France,
and return of outputs to NCSA.
DM-NCSA-26:
description: Demonstrate readiness of the entire data facility prior to operations.
This includes the final operations rehearsal, which will show operational readiness
to acquire data and for day to day operations of the LDF with Observatory Operations.
This also includes readiness to support LSP and DAC serving data to the broader
community.
DM-NCSA-30:
description: This milestone duplicates DM-NCSA-12; we will be requesting deletion.
DM-NCSA-31:
description: Implement the transformed EFD schema in the consolidated database system.
DM-NCSA-8:
description: Test instance of feeds to LSST mini broker in online (live stream generated
from real-time pipelines) and offline (replaying from files of alert events) modes.
DM-NCSA-9:
description: 'Hosting an instance of the alert distribution service delivered from
University of Washington on our test containerized application management service
platform, running feeds from the pipeline to the distribution services. Note:
This was informally tied to LDM-503-05, which was de-scoped to a small demonstration
of running an instance of Kafka at NCSA, so no L1 database was needed. We believe
this was completed in August 2018.'
DM-NET-4:
description: Installed the Base LAN equipment in the Base Date Center and Base Office
Building technical closets, and cabling connecting them for the Base Campus and
Control Networks. Tested with data volumes and access points equivalent to operational
staffing and control servers for ATS observing. (Level 2 equivalent of Level 3
milestone IT-725-M)
DM-NET-5:
description: All segments of the Base - Archive Network, including La - Serena -
Santiago - Sao Paolo - Miami/Boca Raton - Chicago - Champaign are operationally
available at 100 Gbps line rate (effective rate is typically lower, ~ 80%).
DM-SQRE-2:
description: Members of the integration and commissioning team are able to develop
notebooks in nublado that are capable of the interactive data exploration that
is suitable to their needs to support ComCam commissioning. Data is available
via a Butler to a mounted filesystem. They are able to work effectively, though
in some cases their user experience can be improved. They are primarily trained
in person by bootcamps and may require extensive support via Slack. The service
may be intermittently unavailable due to engineering work.
DM-SQRE-3:
description: Members of the integration and commissioning team as well as science
pipelines developers use nublado to support their interactive data exploration
needs relating to ComCam commissioning. They are able to work effectively though
in some cases their user experience can be improved. Access to EFD data is easily
available from notebooks. Data is available via Butler to a mounted filesystem.
There is a curated library of notebooks modeling common tasks and a way to automatically
verify that they are in working condition. Users can refer to documentation for
answers to common problems. The service is stable and interruptions to the service
are primarily notified well in advance and co-ordinated with the commissioning
team to minimize disruption to the extent that the underlying infrastructure permits
it
DM-SQRE-4:
description: Science pipeline developers and members of the verification and validation
science team use nublado to support their interactive data exploration needs relating
to ComCam commissioning. They are able to work effectively though in some cases
their user experience can be improved. As well as access to EFD, catalogue and
pixel data, they have access to any available APIs to the workflow/batch system
for non-trivial data processing. There are curated libraries of continuously
integrated notebooks modeling common tasks and they is a reasonable workflow to
allow Github-based collaboration on those notebooks. Users can refer to documentation
for how to get started and for answers to common problems. The service is stable
with minimal interruptions, but may not always scale well to large number of users.
DM-SQRE-5:
description: Members of the science collaborations or other non-project staff with
access to data use nublado to support their interactive scientific data exploration.
They are able to share data with each other using LSP sharing mechanisms and can
access data without a direct filesystem mount (eg VO services, remote Butler etc).
The service scales well to large number of users provided the infrastructure is
sized for them. Users can primarily rely on documentation for support, and use
a helpdesk system for questions. There is robust functionality for performing
large distributed catalog operations and for visually exploring LSSTCam sized
images.
DM-STAFF:
description: "DM Management evaluate current DM staffing against project schedule\
\ and budget. If there is a shortfall through mid-2022 (the \u201Cops gap\u201D\
), this is the point at which we need to fill it."
DM-SUIT-10:
description: 'Portal software is adapted for deployment in Kubernetes. The set of
container images required for deployment is documented in an LTD-style manner.
Configuration files for Kubernetes are provided in Github and documented to a
level that permits deployments to be performed by non-IPAC staff.
(NB: Still achievable post-DM-10)'
DM-SUIT-18:
description: "Portal has a specialized search form for image metadata, allowing\
\ interaction with a CAOM2-based data model. Image metadata searches can easily\
\ be connected to image cutout requests. A search form for mosaics is provided.\
\ Cutouts and mosaics are obtained from a DAX-provided SODA service. \n(NB:\
\ Post-DM-10 no additional capabilities beyond DM-SUIT-5 can be provided.)"
DM-SUIT-5:
description: 'Portal provides access to a collection of public-release HSC data
processed by LSST DM, including calibrated single-epoch images, coadds, and catalogs
(including at least an Object-like catalog). Access is obtained via IVOA interfaces
including TAP for catalogs and ObsTAP for image metadata.
(NB: Still achievable post-DM-10)'
DM-SUIT-6:
description: 'Portal provides an interface for the display of a full focal plane''s
worth of image data from a single visit. Access to image metadata is obtained
via IVOA interfaces, either ObsTAP or SIAv2, with additional search capabilities
supporting the identification of a full visit''s worth of CCD-level images. Some
form of preview image for the full focal plane is provided. Ability to keep up
with real-time image acquisition rates is not required in this milestone.
(NB: Post-DM-10 no _specific_ search form for visit images will be provided, but
the "constraints form" part of the (Obs)TAP query interface will permit constructing
this query as long as the visit ID is in the ObsTAP table. No preview image for
the full focal plane will be available without additional work elsewhere in DM,
but if it is made available in ObsTAP the Portal will be able to display it.)'
DM-SUIT-7:
description: 'Portal is integrated with NCSA-provided federated-identity authentication
and authorization system: the Portal recognizes that a user is logged in and displays
that state, and passes the user''s identity along to LSST data services that it
accesses.
(NB: Still achievable post-DM-10)
Claimed in May, based on installation on lsst-lsp-int.'
DM-SUIT-8:
description: 'Portal is integrated with LSST-provided WebDAV or VOSpace service
exposing the User File Workspace. Portal provides functions for saving image and
catalog query results to the logged-in user''s workspace, with the user''s identity
correctly applied. Portal provides functions for accessing data in the user''s
workspace and displaying it using the Portal''s image and table visualization
tools, as well as using data from the workspace in other Portal contexts where
a file can be uploaded, e.g., to perform a multi-object query.
(NB: Still achievable post-DM-10, though only with WebDAV.)'
DM-SUIT-9:
description: 'Portal has the ability to display a spectral image, i.e., one with
a mixed spatial and wavelength WCS, and provide appropriate coordinate and wavelength
readouts associated with the cursor position. Ideally this would be demonstrated
with spectral data from the LATISS instrument on the Auxiliary Telescope.
(NB: Post-DM-10 this will not be supported on LSST resources; however by early
2020 we expect a variety of standard spectral-image FITS WCSes will be supported
in Firefly due to ongoing IRSA development.)'
DMTC-4500-0010:
description: Time-based milestone marking the start of NCSA's Spring 2018 development
cycle.
DMTC-8000-0870:
description: 'Contract Deliverable Report #8 Report on functional fiber, including
uptime, 1st year functional milestone - Complete. Covers first year of operational
service of La Serena - Santiago primary fiber, including availability and outage
information.'
DMTC-8000-0950:
description: 'Contract Deliverable Report #10 Technical Implementation Plan for
40Gbps diverse path - Complete. Covers the fiber and equipment, ownership/service
levels for La Serena - Santiago secondary path.'
DMTC-8000-0990:
description: 'Contract Deliverable Report #11 Full capacity La Serena to Santiago
over Primary link AND Secondary link - Complete. Covers the results of testing
of the La Serena - Santiago primary and secondary paths.'
DMTC-8100-2170:
description: 'Contract Deliverable Report #6 100G managed ring Contract Delivery
#6 Complete. Covers availability and outages.'
DMTC-8100-2210:
description: 'Contract Deliverable Report #7 100G managed ring Contract Delivery
#7 Complete. Covers availability and outages.'
DMTC-8100-2370:
description: Level of effort management task, annual report delivered, covering
all activities, meetings, conferences/workshops, circuits, tools, tests, staffing,
budget/expenditures.
DMTC-8100-2410:
description: Level of effort management task, annual report delivered, covering
all activities, meetings, conferences/workshops, circuits, tools, tests, staffing,
budget/expenditures.
DMTC-8100-2450:
description: Level of effort management task, annual report delivered, covering
all activities, meetings, conferences/workshops, circuits, tools, tests, staffing,
budget/expenditures.
DMTC-8100-2530:
description: 'Contract Deliverable Report #4 Spectrum Complete. Test of first data
passing over Santiago - Boca Raton Spectrum link'
DMTC-8200-0600:
description: ESNet Atlanta - Chicago 2 x 20 Gbps path diverse circuits available
and tested at capacity.
DMTC-8200-6020:
description: ESNet Atlanta - Chicago 2 x 100 Gbps path diverse circuits available
and tested at capacity.
F19F:
description: Time-based milestone marking the completion of the Fall 2019 development
cycle.
F20F:
description: Time-based milestone marking the completion of the Fall 2020 development
cycle.
F20S:
description: Time-based milestone marking the start of the Fall 2020 development
cycle.
F21F:
description: Time-based milestone marking the completion of the Fall 2021 development
cycle.
IT-716-M2:
description: Externally linked to TS milestone for installation of furniture of
Base Office Building and Base Data Center. All cabling to connect offices to
Base LAN has been received.
IT-718-M:
description: Externally linked to TS milestone for full occupancy of Base Data Center,
including all construction, utilities, access and dust control.
IT-725-M:
description: Installed the Base LAN equipment in the Base Data Center and Base Office
Building technical closets, and cabling connecting them for the Base Campus and
Control Networks. Tested with data volumes and access points equivalent to operational
staffing and control servers for ATS observing. (Level 3 equivalent of Level 2
milestone DM-NET-4)
LDM-503-01:
description: This test demonstrates the deployment of the Portal and API Aspects
of the LSST Science Platform (LSE-319, LDM-542, LDM-554) to serve all-sky
precursor data, including catalogs totaling of order 50--100 billion rows, in
a prototype LSST Data Access Center hosted in the LSST Data Facility at NCSA.
test_spec: LDM-540 v1.0 \S3.1 (LSP-00)
LDM-503-02:
description: This test demonstrates the successful execution of a prototype Data
Release Production science payload (LDM-148, LDM-151), processing data
from precursor surveys at a relatively small scale using compute resources at
the LSST Data Facility at NCSA.
test_spec: LDM-534 v3.1 \S3.1 (DRP-00)
LDM-503-03:
description: This test demonstrates the successful execution of a prototype Alert
Generation science payload (LDM-148, LDM-151), processing data from precursor
surveys at a relatively small scale using compute resources at the LSST Data Facility
at NCSA.
test_spec: LDM-533 v1.1 \S3.1 (AG-00)
LDM-503-04:
description: This test demonstrates that LSST data acquisition system can capture
raw data from instrumentation and construct a well-formed image with proper headers.
test_spec: LDM-538 v1.0 \S4.1 (RAS-00-00)
LDM-503-04b:
comment: The archive or permanent record of the survey includes site filesystems
with multiple copies, immutable data and entries in the consolidated database
for later retrieval and search needs.
description: This test demonstrates that the well-formed image generated in LDM-503-04
can be written to the permanent record of the survey.
short_name: AuxTel DAQ Interface Verification and Spectrograph Ops Rehearsal
test_spec: LDM-538 v1.0 \S4.4 (RAS-00-20)
LDM-503-05:
description: This test demonstrates that a full-LSST-scale (10,000 messages every
39 seconds) alert stream can be distributed to end users. It further shows that
simple filters based on alert contents can be applied to that stream.
test_spec: LDM-533 v2.0 \S\S 4.7--4.9 (test cases LVV-T216, LVV-T217, LVV-T218)
LDM-503-06:
comment: The test will include transferring raft-scale data from the Base Data Center
to NCSA, and ingestion into both Data Back Bone and Butler infrastructures. The
DBB will be tested to ensure that no data loss can occur (for example, because
the data is replicated to multiple locations). The database management system
that houses the Butler and DBB registries will also be tested to demonstrate that
it can recover in the event of data being lost.
description: This milestone records successful transfer of an image equivalent to
one raft from the DAQ at the summit to reliable storage in the LSST Data Facility
at NCSA, from where it will be made available for scientific evaluation through
the LSST Science Platform.
jira: DM-17119
jira_testplan: DM-16074
test_spec: DMTR-171
LDM-503-07:
comment: ''
description: This test demonstrates that the LSST Science Pipelines can be successfully
used to load and perform basic processing of data from the LSST Camera test systems.
In the process, it establishes that data is properly made available at the LSST
Data Facility at NCSA, and is accessible through the LSST Science Platform.
jira_testplan: DM-16072
test_spec: DMTR-112
LDM-503-08:
comment: As the capabilities of the spectrograph grow data can be automatically
ingested into the Data Backbone for permanent record of the survey after being
sent to NCSA.
description: This test demonstrates data acquisition from the Auxiliary Telescope
spectrograph, transmission of all image files with associated metadata to NCSA
for screening, and then further processing if needed.
jira: DM-17131
jira_testplan: DM-16455
test_spec: DMTR-121
LDM-503-08b:
comment: The archive or permanent record of the survey includes site filesystems
with multiple copies, immutable data and entries in the consolidated database
for later retrieval and search needs.
description: This test demonstrates the integration of the CCOB with the permanent
record of the survey.
jira: DM-17118
jira_testplan: DM-16073
name: Small Scale CCOB Data Access
test_spec: DMTR-102
LDM-503-09:
description: This milestone represents the completion of a multi-day exercise in
which (simulated) observations are performed and data is processed in as close
to an operational mode as is possible at this stage in construction. In particular,
it will demonstrate transfer, archiving and ingestion of raw data; offline processing
of calibration and science data; and curation of the resultant data products.
jira: DM-17120
jira_testplan: DM-16075
test_spec: LDM-643
LDM-503-09a:
description: This milestone describes a software release of the LSST Science Pipelines
in support of activities in late 2018 and early 2019. The contents of the release
are described in LDM-564. The release will be accompanied by a characterization
report.
jira: DM-17121
jira_testplan: DM-16076
test_spec: DMTR-111
LDM-503-10:
comment: This milestone will take simulated data from the DAQ at the Summit and
use the DWDM network environment to place data on DM machines at Base Data Center
(BDC), essentially extending the DAQ extended network to the BDC. The test machines
at the BDC will be a L1 handoff environment and a single forwarder
description: Test of the extended data acquisition (DAQ) network in Chile from the
Summit to the Base Data Center.
jira: DM-17122
jira_testplan: DM-16193
test_spec: DMTR-181
LDM-503-10a:
comment: Success will be demonstrated on a Kubernetes cluster provided by NCSA. It
is not required for authorization to be applied at the database level; it is sufficient
for this milestone to apply at the TAP level. Data served will remain that from
the original PDAC work, i.e., SDSS Stripe 82 and/or WISE.
description: This test demonstrates the successful integration of a single-sign-on
federated authentication system, and a basic authorization system, with the three
Aspects of the LSST Science Platform (Portal, Notebook, and API), with the API
Aspect containing at least a TAP service.
jira: DM-19209
jira_testplan: DM-19211
name: 'Science Platform: TAP service with federated SSO A&A'
test_spec: DMTR-161
LDM-503-10b:
comment: This is a data transfer of data from SLAC or some other site with 21-raft-sized
images to NCSA, ingest it into a Butler environment and place the file into file
systems readable by the LSP. The CCOB device might NOT be available, but as 21
raft size data will be available from a test stand at SLAC, we will use a generic
test stand data transfer method (e.g., rsync) to bring designated data to NCSA,
ingest it, and place into appropriate filesystems, and make available through
the LSP.
description: Demonstrate the ability to transfer data from the CCOB with 21 rafts
from SLAC and ingested at NCSA and make available through an instance of the LSP
jira: DM-17123
jira_testplan: DM-16194
name: Large Scale CCOB Data Access
test_spec: DMTR-182
LDM-503-11:
comment: This is a test on the L1 test stand at NCSA to verify the image acquisition
and readout from the DAQ to the Archiver at one raft image scale and rate, and
the L1 handoff from the archiver to the
description: "Template generation, including code required to build templates in\
\ both Data Release and Alert Production science payloads, can now product templates\
\ which are corrected for the effects of Differential Chromatic Refraction (DCR),\
\ as described in LDM-151 \xA76.18. Further, noise in image differences can be\
\ \u201Cdecorrelated\u201D, as described in LDM-151 \xA76.19.1. Both of these\
\ represents initial capabilities, providing realistic interfaces, data-flows\
\ and products; ongoing refinement of the algorithms being used is expected."
jira: DM-17124
jira_testplan: DM-16195
test_spec: LDM-643
LDM-503-11a:
comment: This includes both data flow from ComCam to the Data Backbone, and scientific
processing of ComCam data.
description: Demonstrate that all DM functionality required to support ComCam operations
has been delivered.
jira: DM-17125
jira_testplan: DM-16196
test_spec: ''
LDM-503-11b:
description: This milestone describes a software release of the LSST Science Pipelines
in support of activities in late 2019 and early 2020. The contents of the release
are described in LDM-564. The release will be accompanied by a characterization
report.
jira: DM-17126
jira_testplan: DM-16197
test_spec: DMTR-192
LDM-503-12:
comment: ''
description: This milestone builds upon LDM-503-11, and serves to demonstrate that
all necessary procedures and workflows are in place to support commissioning activities.
jira: DM-17127
jira_testplan: DM-16198
test_spec: ''
LDM-503-12a:
comment: This includes both data flow from the Camera to the Data Backbone, and
scientific processing of LSST data.
description: Demonstrate that all DM functionality required to support operations
with the full LSST Camera has been delivered.
jira: DM-17128
jira_testplan: DM-16199
LDM-503-13:
description: This milestone builds upon LDM-503-12, but with its scope extended
beyond procedures which are necessary in commissioning to cover the operational
era.
jira: DM-17130
jira_testplan: DM-16201
short_name: 'Ops Rehearsal for DRP #1'
test_spec: ''
LDM-503-13a:
description: This milestone describes a software release of the LSST Science Pipelines
in support of activities in late 2020 and early 2021. The contents of the release
are described in LDM-564. The release will be accompanied by a characterization
report.
jira: DM-17129
jira_testplan: DM-16200
test_spec: DMTR-302
LDM-503-14:
comment: In addition to delivering operational code and services, it will also be
necessary to deliver the code which demonstrates that the system provided meets
the requirements placed upon it.
description: This milestone represents the point at which all code and/or services
which are supplied by DM in support of science verification will be tested and
demonstrated to meet their requirements.
jira: DM-17133
jira_testplan: DM-17132
test_spec: ''
LDM-503-14a:
description: The Rubin Science Platform is ready to support Data Preview 0.1 as
deployed on the IDF. The RSP capabilities are largely those demonstrated under
previous milestones, but ported to the IDF and with deployment and
authentication/authorization infrastructure improvements that support the scale
of DP0.1 usage. The Notebook Aspect provides access to the Rubin Science Pipelines
stack and to data exposed in a cloud-deployed Gen3 Butler repository; the API Aspect
provides TAP queries against a Qserv deployment of catalog tables, but not image
metadata; and the Portal Aspect provides a UI for visual and ADQL construction of
queries against those tables. Image and image metadata access services in the API
and Portal Aspects are not included.
jira: DM-26349
jira_testplan: DM-26340
test_spec: DMTR-301
short_name: Science Platform ready for DP0.1
LDM-503-15:
description: This milestone builds upon LDM-503-13, and will demonstrate the production
of a data release under simulated operational conditions using data from science
verification activities.
jira: DM-17135
jira_testplan: DM-17134
short_name: 'Ops Rehearsal for DRP #2'
test_spec: ''
LDM-503-15a:
description: This milestone describes a software release of the LSST Science Pipelines
in support of activities in late 2021 and early 2022. The contents of the release