Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Broken metadata error propagation #541

Merged
merged 5 commits into from Oct 11, 2021
Merged

Conversation

JoranAngevaare
Copy link
Member

@JoranAngevaare JoranAngevaare commented Oct 7, 2021

What is the problem / what does the code in this PR do

This is a very simple PR and a small test to go with it for good practice.

Broken metadata (e.g. due to killed jobs) can cause headaches. It's sometimes very unclear where the error comes from as a lot of metadata may be scanned for building the tree of plugins to be processed

Can you briefly describe how it works?

Add more info to the traceback

What is the advantage?

The context can now recognize that something is wrong with the data and can rebuild the data when asked

Test example.

Running the test one can see the difference in errors

Before

json.decoder.JSONDecodeError: Extra data: line 176 column 2 (char 5406)

After

strax.storage.common.DataCorrupted: Cannot open metadata in /tmp/strax_data/0-records-j3nd2fjbiq from Extra data: line 176 column 2 (char 5406)

Copy link
Contributor

@jmosbacher jmosbacher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a very good idea and I like the implementation of moving the actual logic to an internal method and wrapping it with a try, except block in the public method.
Just one question, why not do this in the base class and have all subclasses implement the private method instead of the public one?

@JoranAngevaare
Copy link
Member Author

Looks like a very good idea and I like the implementation of moving the actual logic to an internal method and wrapping it with a try, except block in the public method. Just one question, why not do this in the base class and have all subclasses implement the private method instead of the public one?

Nice idea Yossi, thanks, let me change it!

@JoranAngevaare JoranAngevaare added the enhancement New feature or request label Oct 7, 2021
Copy link
Contributor

@jmosbacher jmosbacher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@JoranAngevaare JoranAngevaare merged commit e8229e4 into master Oct 11, 2021
@JoranAngevaare JoranAngevaare deleted the JoranAngevaare-patch-1 branch October 11, 2021 11:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants