Skip to content
This repository has been archived by the owner on Jul 19, 2023. It is now read-only.

Experimenting with storing stacktraces differently #757

Conversation

simonswine
Copy link
Collaborator

@simonswine simonswine commented Jun 7, 2023

With our given problems around retrieving esp. stacktraces in a reasonable amount of time. I have been experimenting a bit.

Two ideas:

  • Rather than storing the full location ID list us a self referencing schema.
  • Make use of parquet's nested object to keep track of the ids.

I will report back if any of the provide more tangible results.

This is the current size comparision between status quo and the self referencing model:

image

Copy link
Collaborator

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's interesting makes me wonder if we should explore a custom format for storing this.

@simonswine
Copy link
Collaborator Author

Superseeded by #767

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants