Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement auto-purging of storages #87

Closed
B4nan opened this issue Apr 4, 2024 · 1 comment
Closed

Implement auto-purging of storages #87

B4nan opened this issue Apr 4, 2024 · 1 comment
Assignees
@vdusek
Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.
Milestone
89th sprint - Too...

Comments

@B4nan
Copy link
Member

B4nan commented Apr 4, 2024 •
edited
Loading

We need the same bahavior as with the JS version:

  • crawlee implements the base storage classes
  • every async operation checks if it was the first call, and purges automatically unless opted-out via CRAWLEE_PURGE_ON_START env var (with a falsy value like 0 or false)
  • we have this method that it called on many places in the storage methods like open or getInput https://crawlee.dev/api/core/function/purgeDefaultStorages
  • since the SDK uses those storage classes, it has the same behavior out of box
  • internally this works by calling purge method on the storage client, so this also means both memory storage and apify client need to implement this purge method

Related: apify/apify-cli#545
@B4nan B4nan mentioned this issue Apr 4, 2024
Closed
@vdusek vdusek added t-tooling Issues with this label are in the ownership of the tooling team. enhancement New feature or request. labels Apr 4, 2024
@vdusek vdusek self-assigned this Apr 24, 2024
@vdusek vdusek added this to the 88th sprint - Tooling team milestone Apr 24, 2024
@vdusek vdusek mentioned this issue May 13, 2024
Merged
vdusek added a commit that referenced this issue May 13, 2024
@vdusek
fix: storage manager & purging the defaults (#150)
851042f 
### Description

- Auto-purging of default storages was already implemented. I just
checked it how/if it works, how it works with more instances of
storages, and updated the StorageManager to be singleton-like.

### Related issues

- #87

### Testing

- Covered by unit-tests
@vdusek
Copy link
Collaborator

vdusek commented May 13, 2024

Closing as it was resolved in #150

@vdusek vdusek closed this as completed May 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vdusek vdusek

Labels
enhancement New feature or request. t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Milestone
89th sprint - Tooling team
Development

No branches or pull requests
2 participants
@B4nan @vdusek