You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is mainly motivated by the new AdaptivePlaywrightCrawler. If the user calls e.g. Actor.pushData instead of the respective context helper, the functionality falls flat (and the dataset will become corrupted on top of that).
We should find a way to emit a warning when the user does something like this from a request handler when an adaptive crawler is running (for example).
The text was updated successfully, but these errors were encountered:
Add a checkStorageWrite helper to @crawlee/utils, plus an AsyncLocalStorage instance (e.g. storageWriteChecker) that contains a callback invoked by checkStorageWrite
When a request handler is called from AdaptivePlaywrightCrawler, wrap it in storageWriteChecker.run with a callback that just raises an exception - this prevents request handlers from writing into storage directly
commitResult is called outside of the storageWriteChecker
a checkStorageWrite call needs to be added to each method of MemoryStorage and ApifyClient
or this could be done in a more dynamic fashion (monkey patching) somewhere in StorageManager
just checking in storages.Dataset and the like should be enough, too, and it's probably the easiest option
This is mainly motivated by the new
AdaptivePlaywrightCrawler
. If the user calls e.g.Actor.pushData
instead of the respective context helper, the functionality falls flat (and the dataset will become corrupted on top of that).We should find a way to emit a warning when the user does something like this from a request handler when an adaptive crawler is running (for example).
The text was updated successfully, but these errors were encountered: