Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: clarify AWS Lambda storage #2477

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

docs: clarify AWS Lambda storage #2477

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fab0e3c
Clarify AWS Lambda storage
connorads May 20, 2024
54d3217
Try to add whitespace back
connorads May 20, 2024
47d9398
Trying to resolve diff
connorads May 20, 2024
d1c4ef4
Fix whitespace hopefully
connorads May 20, 2024
70a4fdd
Put underlying reason for not using file storage
connorads May 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions website/versioned_docs/version-3.10/deployment/aws-cheerio.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Locally, we can conveniently create a Crawlee project with `npx crawlee create`.

Whenever we instantiate a new crawler, we have to pass a unique `Configuration` instance to it. By default, all the Crawlee crawler instances share the same storage - this can be convenient, but would also cause “statefulness” of our Lambda, which would lead to hard-to-debug problems.

Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, as the Lambda filesystem is read-only.
Also, when creating this Configuration instance, make sure to pass the `persistStorage: false` option. This tells Crawlee to use in-memory storage, which will also prevent Lambda "statefulness" and its potential issues.

```javascript title="src/main.js"
// For more information, see https://crawlee.dev/
Expand Down Expand Up @@ -123,4 +123,4 @@ The memory size can greatly affect the execution speed of your Lambda.

See the [official documentation](https://docs.aws.amazon.com/lambda/latest/operatorguide/computing-power.html) to see how the performance and cost scale with more memory.

:::
:::
Loading