Scrapyd has directories like:
/home/collect/scrapyd/jobs/kingfisher/germany_digiwhist/8c630d3ab5ec11f088609c6b004cdfa7/requests.queue
These contain active.json which is just [] for finished jobs.
Since these change frequently for running jobs, they cause an error when syncing:
An error occurred (IncompleteBody) when calling the PutObject operation: You did not provide the number of bytes specified by the Content-Length HTTP header
We should ignore these directories.
Scrapyd has directories like:
These contain
active.jsonwhich is just[]for finished jobs.Since these change frequently for running jobs, they cause an error when syncing:
We should ignore these directories.