DM-32029: Add faro steps to DRP.yaml

[jeffcarlin]

Successful Jenkins run is at https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35175/pipeline/

The blocker (DM-32059) has been completed and merged - that ticket removed the unused faro tasks from the main pipelines so they wouldn't cause problems in obs_subaru.

[arunkannawadi]

On the commits themselves - why are there two sets of commits with identical messages? There probably shouldn't be the merge commit either. And commit messages should not end in a period

[arunkannawadi]

Would it make sense to categorize faro tasks as faro_step1, faro_step2 etc. based on when they can be run and faro_step1 can be included in step1? If that is not possible to merge before the weekly deadline, please go ahead and merge but you could consider reorganization later. The reason I ask is that it'd make it easier to add new metrics/tasks later on as we add them.

[jeffcarlin]

We could do that - the reason the faro subset exists is so that it's easier to exclude the faro steps from ci_hsc_gen3, which returns almost all NaNs from faro tasks. But we could easily exclude multiple faro subsets from ci_hsc_gen3 processing.

I would rather do that reorganization later (it's a good idea!). I think for now this satisfies the goal of integrating faro into the processing workflow.

[jeffcarlin]

On the commits themselves - why are there two sets of commits with identical messages? There probably shouldn't be the merge commit either. And commit messages should not end in a period

I think something got funky when I rebased. I can squash those commit messages before merging.

[natelust]

You cannot use subsets inside other subsets at this time

[jeffcarlin]

Ah, too bad. Thanks for the info, though -- I'll put the explicit task labels back in the pipelines.

[natelust]

Its a common assumption (and a good use case), but when subsets were created we didnt want to deal with the complexity of things like circular references and such. I have some ideas about this, but for now they cant be used, sorry!

[jeffcarlin]

Totally makes sense. That could easily become circular...

[natelust]

I think some of the faro task labels are missing from the subsets, please make sure all the tasks defined by faro appear inside a subset, otherwise unintended tasks and or failures will occur.

[jeffcarlin]

I think some of the faro task labels are missing from the subsets, please make sure all the tasks defined by faro appear inside a subset, otherwise unintended tasks and or failures will occur.

This should be fixed now. I removed all the superfluous tasks from the default faro pipelines in DM-32059.

[arunkannawadi]

You cannot use subsets inside other subsets at this time

Ah, this was the part I was unsure about and wanted Jeff to check. Good to know that this isn't possible right now.

[arunkannawadi]

What you have seems to be a good compromise between what is possible and what would have been ideal (but impossible). And you seem to have picked up some other version of my commits from yesterday which are already merged onto master. Have you rebased it correctly? Please checkout master branch, pull all the changes, and rebase this ticket branch onto the latest master. Drop the merge commits and any other commits that have to do with GAaP