Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Track time spent in the FetcherBolt queues #535
Knowing how long a URL has spent in the queues is very useful and a good addition to knowing how long the actual fetching took. This could be used for instance to throttle the emissions of URLs from a particular backend, similar to #504. These also reflect the robots wait directives - as well as the actual waiting time for the queue.
This should also be provided as metrics.