DM-40414: Write PipelineTask to transmit alerts to Kafka #191

bsmartradio · 2024-02-01T00:45:18Z

Add a producer to packageAlerts in ap_association. The producer will send alerts from the pipeline to the alert stream using several environment variables to set the server, user, password, and topic.

Jenkins run here: https://rubin-ci.slac.stanford.edu/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/571/pipeline

ebellm

looks good overall, a range of suggestions throughout.

ebellm · 2024-02-02T17:08:45Z

python/lsst/ap/association/packageAlerts.py

+    from confluent_kafka import KafkaException
+    import confluent_kafka
+except ImportError as error:
+    error.msg += ("Could not import confluent_kafka. Alerts will not be sent to the alert stream")


This message isn't logged anywhere...

I think it might be more appropriate to use error.add_note("Could not import...") rather than trying to modify the original error message.

Moved this so I could add logging.

ebellm · 2024-02-02T17:11:33Z

python/lsst/ap/association/packageAlerts.py


 """Methods for packaging Apdb and Pipelines data into Avro alerts.
 """
+_ConfluentWireFormatHeader = struct.Struct(">bi")
+latest_schema = Schema.from_file().definition


Don't load this here--you want to use the schemaFile set up in PackageAlertsConfig.

Moved _ConfluentWireFormatHeader and swapped to schemaFile

ebellm · 2024-02-02T17:25:00Z

python/lsst/ap/association/packageAlerts.py

+        topic = self.kafka_topic
+
+        for alert in alerts:
+            alert_bytes = self._serialize_alert(alert, schema=latest_schema, schema_id=1)


Suggested change

alert_bytes = self._serialize_alert(alert, schema=latest_schema, schema_id=1)

alert_bytes = self._serialize_alert(alert, schema=self.alertSchema.definition, schema_id=1)

ebellm · 2024-02-02T17:25:30Z

python/lsst/ap/association/packageAlerts.py

@@ -372,3 +451,76 @@ def streamCcdDataToBytes(self, cutout):
            cutout.write(streamer, format="fits")
            cutoutBytes = streamer.getvalue()
        return cutoutBytes
+
+    def _serialize_alert(self, alert, schema=latest_schema, schema_id=0):


Suggested change

def _serialize_alert(self, alert, schema=latest_schema, schema_id=0):

def _serialize_alert(self, alert, schema=self.alertSchema.definition, schema_id=0):

ebellm · 2024-02-02T17:29:36Z

python/lsst/ap/association/packageAlerts.py

@@ -63,6 +77,12 @@ class PackageAlertsConfig(pexConfig.Config):
        default=os.path.join(os.getcwd(), "alerts"),
    )

+    produceAlertsConfig = pexConfig.Field(


Suggested change

produceAlertsConfig = pexConfig.Field(

doProduceAlerts = pexConfig.Field(

Swapped to this and added doWriteAlerts

ebellm · 2024-02-02T17:30:55Z

python/lsst/ap/association/packageAlerts.py

+        if self.config.produceAlertsConfig and "confluent_kafka" in sys.modules:
+            self.produceAlerts(alerts, ccdVisitId)
+
+        else:


I'd prefer that we use separate configs for producing and writing alerts: doProduceAlerts and doWriteAlerts. Then we can raise an error if doProduceAlerts is set and confluent_kafka isn't loaded.

As mentioned above, we now have both, as well as a check if neither is called set since that would be a situation where nothing happens to the alerts.

ebellm · 2024-02-02T17:49:35Z

python/lsst/ap/association/packageAlerts.py

+            Format prefix.
+        """
+        buf = io.BytesIO()
+        # TODO: Use a proper schema versioning system


Suggested change

# TODO: Use a proper schema versioning system

# TODO: Use a proper schema versioning system (DM-42606)

ebellm · 2024-02-02T17:52:02Z

tests/test_packageAlerts.py

+try:
+    import confluent_kafka
+except ImportError as error:
+    error.msg += ("Could not import confluent_kafka. Alerts will not be sent to the alert stream")


Again, this error message isn't being used.

The import has been moved inside the __init__ to use logging.

The import now is back with the main imports and sets confluent_kafka = None when it fails. This then triggers an error message to be logged if doProduceAlerts was set.

ebellm · 2024-02-02T17:55:18Z

tests/test_packageAlerts.py

We should test applying and interpreting the Confluent wire format header.

Test added as test_serialize_alert_round_trip. As per our discussion, there is some concern on the loss of float accuracy during the round trip, as it can vary between 5 and 7 decimal points.

Test is now fixed with setting variables explicitly to 32 bit floats.

ebellm · 2024-02-02T17:55:53Z

python/lsst/ap/association/packageAlerts.py


 """Methods for packaging Apdb and Pipelines data into Avro alerts.
 """
+_ConfluentWireFormatHeader = struct.Struct(">bi")


Does this really need to be a global? Can it live in PackageAlertsTask?

This now lives in the PackageAlertTask _serialize_confluent_wire_header and _deserialize since it isn't needed elsewhere.

parejkoj

I've made a lot of cleanup comments. I didn't look at all of the test: make the skipIf decorator cleanups and I'll look at the tests more.

You have a mix of snake_case and camelCase in the changes here: please be consistent in this package. The existing code is camelCase, so please use that for all variables and functions.

python/lsst/ap/association/diaPipe.py

python/lsst/ap/association/packageAlerts.py

parejkoj · 2024-02-15T21:23:44Z

python/lsst/ap/association/packageAlerts.py

@@ -63,6 +69,18 @@ class PackageAlertsConfig(pexConfig.Config):
        default=os.path.join(os.getcwd(), "alerts"),
    )

+    doProduceAlerts = pexConfig.Field(
+        dtype=bool,
+        doc="Turn on alert production to kafka if true. Set to false by default",


Don't repeat the default in the docstring, it will be displayed in the docs automatically. Same for the below.

Suggested change

doc="Turn on alert production to kafka if true. Set to false by default",

doc="Turn on alert production to kafka if true.",

parejkoj · 2024-02-15T21:35:10Z

python/lsst/ap/association/packageAlerts.py

+
+    doWriteAlerts = pexConfig.Field(
+        dtype=bool,
+        doc="Write alerts to disk if true. Set to true by default",


How does this config interact with doProduceAlerts above? This suggests we're just writing to disk, the other suggests we're sending to kafka: are they exclusive, or can you do both? That should go in these docs.

They are both independent of one another, and can both be set or one or the other. An error only needs to be set if both are not set.

parejkoj · 2024-02-15T22:07:49Z

python/lsst/ap/association/packageAlerts.py

+
+    def _delivery_callback(self, err, msg):
+        if err:
+            _log.debug('%% Message failed delivery: %s\n' % err)


If it failed, shouldn't it be log.error here or log.warning?

tests/test_packageAlerts.py

parejkoj

How do we run the tests that depend on kafka, if kafka isn't in rubin-env? Is it in rubin-env-developer?

bsmartradio · 2024-02-15T22:44:37Z

It is in my rubin-dvl, I cannot remember if I set that up or if its present for everyone.

…

________________________________ From: John Parejko ***@***.***> Sent: Thursday, February 15, 2024 2:42 PM To: lsst/ap_association ***@***.***> Cc: Brianna Smart ***@***.***>; Author ***@***.***> Subject: Re: [lsst/ap_association] DM-40414: Write PipelineTask to transmit alerts to Kafka (PR #191) @parejkoj commented on this pull request. How do we run the tests that depend on kafka, if kafka isn't in rubin-env? Is it in rubin-env-developer? — Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/lsst/ap_association/pull/191*pullrequestreview-1884014119__;Iw!!K-Hz7m0Vt54!jhDxj_ehNOo0DRtVq2DAMt5FXFWfxqHTHa9lyahcpNzkv5CZTaYndjBuKUCe-7xV0mrgRDK3qiXE_riGvlnQeRM$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AWZRE6BYC6DCAKNJE6VD6NLYT2FONAVCNFSM6AAAAABCT7ZCCKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQOBUGAYTIMJRHE__;!!K-Hz7m0Vt54!jhDxj_ehNOo0DRtVq2DAMt5FXFWfxqHTHa9lyahcpNzkv5CZTaYndjBuKUCe-7xV0mrgRDK3qiXE_riGV7J1Lps$>. You are receiving this because you authored the thread.Message ID: ***@***.***>

ebellm

Getting there! A couple more small items.

ebellm · 2024-02-17T00:26:18Z

python/lsst/ap/association/packageAlerts.py

+    doWriteAlerts = pexConfig.Field(
+        dtype=bool,
+        doc="Write alerts to disk if true.",
+        default=True,


Suggested change

default=True,

default=False,

This should default to false to match the existing behavior.

Also, I just appreciated that we need to formally deprecate doPackageAlerts rather than simply removing it: see https://developer.lsst.io/stack/deprecating-interfaces.html#config-deprecation

also needs an RFC: https://developer.lsst.io/communications/rfc.html#standard-procedures-that-require-an-rfc

If we wanted to avoid the song and dance we could just continue to use doPackageAlerts in place of doWriteAlerts. It's less clear but less hassle.

I'll go this way for now.

Looking at the package again, I think we can actually leave doWriteAlerts as doPackageAlerts is actually at the diaPipe level and not in the PackageAlertsTask? Do doPackageAlerts gets set in dia pipe, then inside there doWriteAlerts or doProduceAlerts gets set. There was never a config to turn on or off writing the alerts out.

python/lsst/ap/association/packageAlerts.py

ebellm · 2024-02-17T00:33:17Z

tests/test_packageAlerts.py

+
+        header_bytes = alert_bytes[:5]
+        version = self._deserialize_confluent_wire_header(header_bytes)
+        assert version == 0


self.assertEqual instead?

Missed changing that one, thanks for pointing it out. Swapped to self.assertEqual

I think the purpose of that bare assert was to check that the version coming out of the header is always 0. I don't know if that's actually the behavior we want here, but I don't think it's a unittest-related thing, but rather a "is the code sensible" test, hence a bare assert. I might be wrong, though?

parejkoj · 2024-02-21T00:58:01Z

python/lsst/ap/association/diaPipe.py

+            try:
+                self.alertPackager.run(associatedDiaSources,
+                                       diaCalResult.diaObjectCat,
+                                       loaderResult.diaSources,
+                                       diaForcedSources,
+                                       diffIm,
+                                       template)
+            except ValueError as err:
+                # Continue processing even if alert sending fails
+                self.log.error(err)


The task no longer raises ValueError in run: I think you should probably just call run without a try: block here (aka, remove your change), and we'll sort out what error conditions we might want to handle with a post RFC-958 custom error. As it stands, you're already catching and logging kafka errors, which I think is all that we should do for error handling here, until proven otherwise.

Alright, removed

python/lsst/ap/association/packageAlerts.py

parejkoj · 2024-02-21T01:00:12Z

python/lsst/ap/association/packageAlerts.py

+                self.log.error("Produce alerts is set but confluent_kafka is not present in "
+                               "the environment. Alerts will not be sent to the alert stream.")


Shouldn't this be a raise (probably RuntimeError for now?)? If someone sets that to True, they want to produce alerts.

I assume then it would be fine to remove the self.log.error since raise would be logged?

Yes, if we just raise we don't need to log.

python/lsst/ap/association/packageAlerts.py

parejkoj · 2024-02-21T01:01:44Z

python/lsst/ap/association/packageAlerts.py

+                self.password = os.getenv("AP_KAFKA_PRODUCER_PASSWORD")
+                self.username = os.getenv("AP_KAFKA_PRODUCER_USERNAME")
+                self.server = os.getenv("AP_KAFKA_SERVER")
+                self.kafkaTopic = os.getenv("AP_KAFKA_TOPIC")
+
+                if not self.password:
+                    raise ValueError("Kafka password environment variable was not set.")
+                if not self.username:
+                    raise ValueError("Kafka username environment variable was not set.")
+                if not self.server:
+                    raise ValueError("Kafka server environment variable was not set.")
+                if not self.kafkaTopic:
+                    raise ValueError("Kafka topic environment variable was not set.")


Since you can't do these with a walrus, at least put each if directly under the value that is being assigned to.

parejkoj · 2024-02-21T01:15:23Z

python/lsst/ap/association/packageAlerts.py

+            self.log.debug('%% Message delivered to %s [%d] @ %d\n'
+                           % (msg.topic(), msg.partition(), msg.offset()))


Use lazy formatting:

Suggested change

self.log.debug('%% Message delivered to %s [%d] @ %d\n'

% (msg.topic(), msg.partition(), msg.offset()))

self.log.debug('%% Message delivered to %s [%d] @ %d', msg.topic(), msg.partition(), msg.offset())

parejkoj · 2024-02-21T01:15:52Z

tests/test_packageAlerts.py

+    def __init__(self, *args, **kwargs):
+        super(TestPackageAlerts, self).__init__(*args, **kwargs)
+


Unnecessary, please remove.

parejkoj · 2024-02-21T01:16:42Z

tests/test_packageAlerts.py

+
+        header_bytes = alert_bytes[:5]
+        version = self._deserialize_confluent_wire_header(header_bytes)
+        self.assertEqual(version, 0)


What is the purpose of this assert?

Its a sanity check leftover to make sure the version was correct. I can remove it since it isn't necessary for the unit checks.

parejkoj · 2024-02-21T01:17:06Z

tests/test_packageAlerts.py

@@ -153,6 +204,49 @@ def setUp(self):
        self.cutoutWcs.wcs.cd = self.exposure.getWcs().getCdMatrix()
        self.cutoutWcs.wcs.ctype = ["RA---TAN", "DEC--TAN"]

+    def _deserialize_alert(self, alert_bytes):


I think this, and the _deserialize_confluent_wire_header should both be free functions here: they don't need to live in the test class at all.

I've removed _deserialized_confluent_wire_header because its not needed if we aren't doing the header version sanity check.

parejkoj · 2024-02-21T01:31:15Z

tests/test_packageAlerts.py

+
+        header_bytes = alert_bytes[:5]
+        version = self._deserialize_confluent_wire_header(header_bytes)
+        assert version == 0


I think the purpose of that bare assert was to check that the version coming out of the header is always 0. I don't know if that's actually the behavior we want here, but I don't think it's a unittest-related thing, but rather a "is the code sensible" test, hence a bare assert. I might be wrong, though?

parejkoj

More changes.

parejkoj · 2024-02-21T03:23:30Z

There are some comments from the previous review that were apparently missed: I marked them with 👀 .

ebellm · 2024-02-23T19:14:34Z

python/lsst/ap/association/packageAlerts.py

@@ -63,6 +67,18 @@ class PackageAlertsConfig(pexConfig.Config):
        default=os.path.join(os.getcwd(), "alerts"),


I view this behavior as only meant for debugging, for which the cwd is a reasonable default. There's no sensible choice in production as the PP pod will get blown away. (And for that reason we should figure out a more useful scheme on a future ticket.)

Add confluent_kafka to ap_association so that alerts can be sent to the alert stream when doProduceAlerts is set. Additionally, update the write alerts function to only write when doWriteAlerts is set. Add unit tests to test the new functionality.

The schema is f32, so we have to use that in the test data to make it round trip. The assert in the test itself needs to be exact equality.

bsmartradio · 2024-02-28T18:55:29Z

Jenkins run: https://rubin-ci.slac.stanford.edu/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/774/pipeline

parejkoj

A couple remaining comments about docstrings, but otherwise this looks much better, thank you.

parejkoj · 2024-02-29T06:48:20Z

python/lsst/ap/association/packageAlerts.py

+                Parameters
+        ----------


These aren't lined up.

parejkoj · 2024-02-29T06:48:47Z

python/lsst/ap/association/packageAlerts.py

+        """Serialize an alert to a byte sequence for sending to Kafka.
+
+        Parameters
+        ----------`


Excess ` at the end

If people are interested, I have been experimenting with numpydoc validation in middleware packages and can point people at that. Also, the velin package can fix some problems automatically.

@timj Yes I definitely would be interested in that!

velin fixes automatically both of the problems that @parejkoj points out above. It also fixes some other indent issues and variable name and section headings. We don't install velin in the shared stack by default (it's not in rubin-develop-env but maybe it could be cc/ @ktlim ).

numpydoc is also not in the rubin env but has many more extensive checks. It also needs to have a lot of configuration. The quickest way to see what you need to do is to look at lsst/daf_butler#922 and see the changes I made to pyproject.toml and the github workflows.

You can run the validation with:

$ python -m numpydoc.hooks.validate_docstrings $(find python -name "*.py")

after you've installed numpydoc. This will definitely need some tuning in your config files to keep the checks you think are useful.

Moved functions in unit tests and cleaned up packageAlerts.

bsmartradio requested a review from ebellm February 1, 2024 00:45

ebellm requested changes Feb 2, 2024

View reviewed changes

bsmartradio force-pushed the tickets/DM-40414 branch 3 times, most recently from 65493e9 to e7a419f Compare February 13, 2024 20:07

bsmartradio force-pushed the tickets/DM-40414 branch 2 times, most recently from 64a723f to ca76522 Compare February 15, 2024 20:35

parejkoj requested changes Feb 15, 2024

View reviewed changes

parejkoj reviewed Feb 15, 2024

View reviewed changes

bsmartradio force-pushed the tickets/DM-40414 branch 3 times, most recently from 8378085 to 2b06795 Compare February 16, 2024 23:46

ebellm requested changes Feb 17, 2024

View reviewed changes

bsmartradio force-pushed the tickets/DM-40414 branch from 2b06795 to 60552ea Compare February 20, 2024 19:43

parejkoj reviewed Feb 21, 2024

View reviewed changes

parejkoj requested changes Feb 21, 2024

View reviewed changes

ebellm self-requested a review February 27, 2024 20:51

ebellm approved these changes Feb 27, 2024

View reviewed changes

bsmartradio force-pushed the tickets/DM-40414 branch from 32e03c8 to c996371 Compare February 27, 2024 23:15

bsmartradio and others added 2 commits February 28, 2024 10:50

Fix floating point round tripping

2270892

The schema is f32, so we have to use that in the test data to make it round trip. The assert in the test itself needs to be exact equality.

bsmartradio force-pushed the tickets/DM-40414 branch from c996371 to c00d5db Compare February 28, 2024 18:50

parejkoj approved these changes Feb 29, 2024

View reviewed changes

Update unit tests and import handling

3861bca

Moved functions in unit tests and cleaned up packageAlerts.

bsmartradio force-pushed the tickets/DM-40414 branch from c00d5db to 3861bca Compare February 29, 2024 18:14

bsmartradio merged commit 61a7c33 into main Mar 1, 2024
2 checks passed

bsmartradio deleted the tickets/DM-40414 branch March 1, 2024 04:53

	alert_bytes = self._serialize_alert(alert, schema=latest_schema, schema_id=1)
	alert_bytes = self._serialize_alert(alert, schema=self.alertSchema.definition, schema_id=1)

	def _serialize_alert(self, alert, schema=latest_schema, schema_id=0):
	def _serialize_alert(self, alert, schema=self.alertSchema.definition, schema_id=0):

	produceAlertsConfig = pexConfig.Field(
	doProduceAlerts = pexConfig.Field(

	# TODO: Use a proper schema versioning system
	# TODO: Use a proper schema versioning system (DM-42606)

	doc="Turn on alert production to kafka if true. Set to false by default",
	doc="Turn on alert production to kafka if true.",

		self.log.error("Produce alerts is set but confluent_kafka is not present in "
		"the environment. Alerts will not be sent to the alert stream.")

		self.log.debug('%% Message delivered to %s [%d] @ %d\n'
		% (msg.topic(), msg.partition(), msg.offset()))

		def __init__(self, args, *kwargs):
		super(TestPackageAlerts, self).__init__(args, *kwargs)

		@@ -63,6 +67,18 @@ class PackageAlertsConfig(pexConfig.Config):
		default=os.path.join(os.getcwd(), "alerts"),

DM-40414: Write PipelineTask to transmit alerts to Kafka #191

DM-40414: Write PipelineTask to transmit alerts to Kafka #191

Conversation

bsmartradio commented Feb 1, 2024

ebellm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsmartradio Feb 13, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parejkoj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parejkoj left a comment

Choose a reason for hiding this comment

bsmartradio commented Feb 15, 2024 via email

ebellm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parejkoj Feb 21, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

parejkoj left a comment

Choose a reason for hiding this comment

parejkoj commented Feb 21, 2024

Choose a reason for hiding this comment

bsmartradio commented Feb 28, 2024

parejkoj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsmartradio Feb 13, 2024 •

edited

parejkoj Feb 21, 2024 •

edited