Skip to content

julien-duponchelle/scrapy-graphite

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

26 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Description

Output scrapy statistics to carbon/graphite

Install

pip install ScrapyGraphite

Configure scrapy project:

Set STATS_CLASS to scrapygraphite.GraphiteStatsCollector in your scrapy project settings.py as follows

STATS_CLASS = 'scrapygraphite.GraphiteStatsCollector'

If graphite carbon is not using default values (localhost:2003), add GRAPHITE_HOST and/or GRAPHITE_PORT.

GRAPHITE_HOST = "localhost"
GRAPHITE_PORT = 2003

Configure graphite:

In graphite you need to add in your storage-aggregation.conf:

[scrapy_min]
pattern = ^scrapy\..*_min$
xFilesFactor = 0.1
aggregationMethod = min

[scrapy_max]
pattern = ^scrapy\..*_max$
xFilesFactor = 0.1
aggregationMethod = max

[scrapy_sum]
pattern = ^scrapy\..*_count$
xFilesFactor = 0.1
aggregationMethod = sum

Changelog

0.2:

  • Scrapy 0.16 compatibility
  • Default configuration

0.1:

  • Initial version

Thanks

Licence

Copyright 2011 Julien Duponchelle

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

About

Output scrapy statistics to graphite/carbon

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages