Skip to content

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

License

Notifications You must be signed in to change notification settings

ChristianYeah/scrapyd-egg-checksum

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

scrapyd-egg-checksum

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

Installation

pip install scrapyd-egg-checksum

Configuration

Locate the scrapyd.conf file and adding line like

listversions_advanced.json = scrapyd_egg_checksum.webservice.ListVersions

under service section, the json resource name depends on you,

Usage

curl http://127.0.0.1:6800/listversions_advanced.json?project=ABC

you will get like

{"node_name": "Christians-Another-MacBook-Pro.local", "status": "ok", "versions": [{"version": "1_0_0", "checksum": "793afec3676f9749d1616f48dd57fe07"}, {"version": "1_0_1", "checksum": "ccb6b8841e2b5443db4bff56924527fa"}]}

Use case

This is for distributed scrapyd clusters, while in single server mode, this is useless.

About

Extension of scrapyd to get egg's md5 checksum for distributed scrapyd

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages