Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

deprecate P_STORAGE_UPLOAD_INTERVAL #651

Closed
nitisht opened this issue Feb 6, 2024 · 2 comments · Fixed by #653
Closed

deprecate P_STORAGE_UPLOAD_INTERVAL #651

nitisht opened this issue Feb 6, 2024 · 2 comments · Fixed by #653
Assignees
@nikhilsinhaparseable
Labels
enhancement New feature or request
Milestone
v0.8.0 Release

Comments

@nitisht
Copy link
Member

nitisht commented Feb 6, 2024

We explored the option of making the Parquet size configurable by making the upload interval configurable using the env P_STORAGE_UPLOAD_INTERVAL, but the problem with this approach is manifold.

  • The data in staging area is now variable and can span hours or days. This makes the query code very complicated.
  • There is not much benefit on the parquet size, because it is difficult to predict the volume of logs.

A better approach would be to add a separate compaction engine that can compact and create more compressed parquet files for historical data. We'll take that up in a separate exercise. For now we need to revert the changes in #616 and also remove the P_STORAGE_UPLOAD_INTERVAL option completely.
@nitisht nitisht added the enhancement New feature or request label Feb 6, 2024
@nitisht nitisht added this to the v0.7.4 Release milestone Feb 6, 2024
@nitisht nitisht mentioned this issue Feb 6, 2024
Closed
3 tasks
@Delta456
Copy link
Contributor

Delta456 commented Feb 6, 2024

I believe the next release which will remove this needs to be v0.8.0.

nikhilsinhaparseable added a commit to nikhilsinhaparseable/parseable that referenced this issue Feb 6, 2024
@nikhilsinhaparseable
fix for parseablehq#651: removed use of env var P_STORAGE_UPLOAD_INTE…
e81c7f4 
…RVAL

added const of 60 secs to be used for local to storage sync
@nikhilsinhaparseable nikhilsinhaparseable mentioned this issue Feb 6, 2024
Merged
3 tasks
@nitisht
Copy link
Member Author

nitisht commented Feb 7, 2024

I believe the next release which will remove this needs to be v0.8.0.

Yes, that is correct.

nitisht pushed a commit that referenced this issue Feb 7, 2024
@nikhilsinhaparseable
fix: removed use of environment var P_STORAGE_UPLOAD_INTERVAL (#653)
fb3fd21 
added const of 60 secs to be used for local to storage sync

fixes #651
@nitisht nitisht changed the title discard P_STORAGE_UPLOAD_INTERVAL deprecate P_STORAGE_UPLOAD_INTERVAL Feb 7, 2024
nikhilsinhaparseable added a commit to nikhilsinhaparseable/parseable that referenced this issue Apr 20, 2024
@nikhilsinhaparseable
fix: removed use of environment var P_STORAGE_UPLOAD_INTERVAL (parsea…
f873fd1 
…blehq#653)

added const of 60 secs to be used for local to storage sync

fixes parseablehq#651
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nikhilsinhaparseable nikhilsinhaparseable

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.8.0 Release
Development

Successfully merging a pull request may close this issue.

fix for #651: removed use of env var P_STORAGE_UPLOAD_INTERVAL nikhilsinhaparseable/parseable
3 participants
@nitisht @Delta456 @nikhilsinhaparseable