Skip to content

Disable task cache for worker lambdas by default, improve logging#342

Merged
twelch merged 10 commits intodevfrom
worker-cache
Sep 6, 2024
Merged

Disable task cache for worker lambdas by default, improve logging#342
twelch merged 10 commits intodevfrom
worker-cache

Conversation

@twelch
Copy link
Copy Markdown
Contributor

@twelch twelch commented Sep 6, 2024

Workers caching their task results increases the number of dynamodb items per report run by significantly. If 1 gp function splits its work over 5 workers, then there's 5 times as many dynamodb items. The current assumption is that the parent gp function will cache their results, and therefore the worker results don't need to be cached. This will cut down on the amount of dynamodb writes.

Requirements

  • Give runLambdaWorker caller control over whether caching or not. Defaults to not cache. Can be turned on for debugging purposes.

ToDo:

  • Add disableServerCache option to geoprocessing handler request and task model
  • Switch to default to not cache
  • refactor task.create to use options object

@twelch twelch marked this pull request as ready for review September 6, 2024 21:02
@twelch twelch changed the title Add support for disabling task cache for worker lambdas Disable task cache for worker lambdas by default, improve logging Sep 6, 2024
@twelch twelch merged commit 04ffd4a into dev Sep 6, 2024
@twelch twelch deleted the worker-cache branch September 6, 2024 21:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant