New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
analytics-publisher is crashing due to a breaking change in binderhub #2392
Labels
Comments
Maybe something like this... event = json.loads(line)["jsonPayload"]
# Account for time when 'message' was nested
if "message" in event:
event.update(json.loads(event["message"]))
del event["message"] in mybinder.org-deploy/images/analytics-publisher/archiver.py Lines 53 to 64 in 9ef1ae2
Got to go for now. |
Seems to work great!
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@yuvipanda wrote:
Inspection gave me that the pod was unhealthy and failing to start.
The following logs were seen
Understanding what caused this
The line of code that fails
mybinder.org-deploy/images/analytics-publisher/archiver.py
Lines 38 to 54 in 9ef1ae2
The configuration of source bucket for prod
mybinder.org-deploy/config/prod.yaml
Lines 201 to 206 in 9ef1ae2
The GCP bucket
The uploaded data are lines of JSON objects, where how each object looks has changed.
Before
where the "jsonPayload.message" field is a string of JSON that would parse like this:
After
With 2022-10-20 - 11:00 and onwards, a new format started showing that didn't include
jsonPayload.message
.Hypothesis concluded
After 2022-10-20 (about 11:00), the
jsonPayload
changed its structure, wherejsonPayload.message
is unpacked directly injsonPayload
it seems.This coincides with #2369, which in turn includes this changes
That in turn comes from jupyterhub/binderhub#1536
Action points
mybinder.org-deploy upgrade succeeded even though analytics-publisher didn't startup #2391
We could make analytics-publisher script adjust to the new format, or if needed, attempt to parse either.
analytics-publisher: fix for breaking binderhub change, refreeze dependencies, modernize Dockerfile #2393
We could make binderhub, using google's logger library, manage to retain the format we used previously somehow.
Using v1 caused a problem bound to happen sooner or later, so working around it would just give us some time at best - and be an effort to do by itself.
I suggest resolving it with adjusting analytics-publisher's parsing of uploaded data in this location
mybinder.org-deploy/images/analytics-publisher/archiver.py
Lines 38 to 54 in 9ef1ae2
The text was updated successfully, but these errors were encountered: