v0.4.8

Changelog

• 521b82e ExtractOnce flag. Init extracted history with payloads url from db.
• f32dd10 Extraction count check in extractor to prevent inertia after request queue stopped, but some requests are still processing.
• fb26937 Extraction history for pipeline.
• fc1aa9f Job and provider metadata extractor from env vars.
• 4cc82cb Load extracted urls from payloads store.
• 732655c Memory collector with visited urls preloading.
• 10ce1e1 Move ExtractOnce flat to extractor config.
• e694e6f Move fnv hash method to media downloader.
• b479aa7 Payload history collection.
• d13c8c1 Payload id from uuid instead of fnv hash (support multiple payloads per url).
• 2b61b9e Remove Reset method from storage interface.
• 8cdc2ff Remove windmill job meta extractor.
• e53a62c Save job and provider to database.
• 0db06e9 Save payload ID to database, remove redundant PayloadID field.
• 12d3626 Set collect storage file name from const, but in constructor.
• 5d815a9 Set colly AllowURLRevisit if Visit once chosen.
• b1afeb6 Update comment for downloader.
• 77d626f Use interface for extractor injected to collector.
• 92367b7 Visit once and extract once settings.