You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Which package is this bug report for? If unsure which one to select, leave blank
@crawlee/memory-storage
Issue description
The setRecord() function in @crawlee/memory-storage/resource-clients/key-value-store.js uses shapeshift to enforce the types of the record object. If a Buffer is passed through this, it can sometimes cause shapeshift's .parse() to hang for upwards of 10 seconds before returning (sometimes it blows the heap and kills Node).
This has caused havoc with many of our spiders after migrating to Crawlee v3.1.3 (from Apify 2.3.2). One symptom is a pegged eventLoop, the other is heap allocation errors that eventually kill the Node main process.
Many but not all of our spiders were affected which leads us to believe it depends heavily on the raw contents of the Buffer as to what pathological behaviour shapeshift exhibits when trying to parse it. Preventing shapeshift from parsing Buffers fixed all the issues immediately.
// Note the `body` and `contentType` vars come from https://crawlee.dev/api/cheerio-crawler/interface/CheerioCrawlingContextimport{v4asuuidv4}from'uuid';constencoding=Buffer.isBuffer(body)&&contentType?.encoding
? contentType.encoding
: 'utf8';constresourceBuffer=Buffer.isBuffer(body)
? body
: Buffer.from(body,encoding);constresourceWal=awaitKeyValueStore.open('wal-resource');awaitresourceWal.setValue(uuidv4(),resourceBuffer,{contentType: 'application/octet-stream'},);
Package version
@crawlee/memory-storage@3.1.2
Node.js version
18.11.0
Operating system
Ubuntu focal
Apify platform
Tick me if you encountered this issue on the Apify platform
I have tested this on the next release
No response
Other context
No response
The text was updated successfully, but these errors were encountered:
Looking at the shapeshift README, I couldn't see a single mention of Buffer anywhere, suggesting it's not a supported or anticipated type. So even if the performance issue was solved, not sure how Buffers would be treated by s.union(s.null, s.string, s.number, s.object({}).passthrough) ?
Closes#1732Closes#1710
Simple fix really, tell shapeshift inputs can be buffers too (I'll also
see if we can add a generic "just check that its an object" in
shapeshift for times like these)
Which package is this bug report for? If unsure which one to select, leave blank
@crawlee/memory-storage
Issue description
The
setRecord()
function in@crawlee/memory-storage/resource-clients/key-value-store.js
uses shapeshift to enforce the types of therecord
object. If a Buffer is passed through this, it can sometimes cause shapeshift's.parse()
to hang for upwards of 10 seconds before returning (sometimes it blows the heap and kills Node).This has caused havoc with many of our spiders after migrating to Crawlee v3.1.3 (from Apify 2.3.2). One symptom is a pegged eventLoop, the other is heap allocation errors that eventually kill the Node main process.
Many but not all of our spiders were affected which leads us to believe it depends heavily on the raw contents of the Buffer as to what pathological behaviour shapeshift exhibits when trying to parse it. Preventing shapeshift from parsing Buffers fixed all the issues immediately.
Our workaround looks like this:
Original source is here:
crawlee/packages/memory-storage/src/resource-clients/key-value-store.ts
Line 211 in 5ff04fa
Code sample
Package version
@crawlee/memory-storage@3.1.2
Node.js version
18.11.0
Operating system
Ubuntu focal
Apify platform
I have tested this on the
next
releaseNo response
Other context
No response
The text was updated successfully, but these errors were encountered: