Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.
Sign upAttach target labels at scrape time #3047
Comments
fabxc
added
component/scraping
dev-2.0
kind/enhancement
priority/P2
labels
Aug 10, 2017
fabxc
closed this
Sep 14, 2017
This comment has been minimized.
This comment has been minimized.
lock
bot
commented
Mar 23, 2019
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
lock
bot
locked and limited conversation to collaborators
Mar 23, 2019
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
fabxc commentedAug 10, 2017
Currently we are wrapping appends to attach target labels – so right before insertion into the storage. That causes us to first allocate label sets when we first see the target and then creating new label sets, which causes memory spikes when many targets appear at once.
If we can pull that logic deeper in, we can directly build the correct label set.
Slightly less nice abstraction, but probably worth it.