Add KingfisherStoreFiles extension to avoid storing files in spiders

[yolile]

closes #277

[jpmckinney]

I think we should also move get_local_file_path_including_filestore, get_local_file_path_excluding_filestore and _get_crawl_path into the extension (and make the necessary changes elsewhere). Basically, no spider should use the os module or interact with the filesystem.

Right now, the new extension sets a directory attribute, but it doesn't use it. When we move these methods, we can replace self.crawler.settings['FILES_STORE'] with that attribute.

The new extension might need to add the path to the item, so that the KingfisherAPI extension doesn't need to re-calculate the path using these methods.

[jpmckinney]

Let's have a follow-up PR (to keep this PR short) to rename save_response_to_disk to build_item_from_response, and save_data_to_disk to build_item.

[jpmckinney]

@yolile To address the issue you identified, can you add to settings.py:

LOG_LEVEL = os.getenv('SCRAPY_LOG_LEVEL', 'INFO')

with a comment that the items' OCDS data is logged at the DEBUG level, which we don't want to see in the log file.

https://docs.scrapy.org/en/latest/topics/settings.html#log-level

[jpmckinney]

Actually, other DEBUG messages are useful (seeing which requests were retried, etc.). Let's instead add a custom log formatter, so that it excludes the OCDS data: https://docs.scrapy.org/en/latest/topics/logging.html#custom-log-formats

https://docs.scrapy.org/en/latest/topics/settings.html#std-setting-LOG_FORMATTER

[yolile]

@jpmckinney I added all your suggestions (changing georgia and digiwhist_base spiders to be able to move get_local_file_path_including_filestore etc methods inside the store extension, FYI the georgia one wasn't working now anyway)
And also added a post_to_api field to the item to no post to kingfisher-process zip and tar files.
Also, I don't store the item if the item has the number attribute as this attribute is used when a big JSON file is sent to kingfisher-process as the behavior was to store only the original data.

[jpmckinney]

I did some refactoring and other cleanup. Looks good!