Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: Support 0.7.0 HTTP API with querying data #36

Closed
1 task done
atimin opened this issue Jul 28, 2022 · 0 comments · Fixed by #37
Closed
1 task done

Feature: Support 0.7.0 HTTP API with querying data #36

atimin opened this issue Jul 28, 2022 · 0 comments · Fixed by #37
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@atimin
Copy link
Member

atimin commented Jul 28, 2022

Current problem

We use Bucket.list to get timestamps of records to read them:

const data = await bucket.list("entry-1", start, stop);
for (const entry of data) {
  const content = await bucket.read("entry-1", entry.ts);
}

The problem is that we can get a big list of records and it takes time for the storage engine to prepare and send it.

Possible solution

The better approach is to iterate the records. This feature was added already in the engine as GET /b/:bucket_name/:entry_name/q method (See reductstore/reductstore#122). So, we can do something like this:

for await (const record in bucket.query("entry-1", start, stop)) {
  consol.log(record.ts, record.size);
  const content = await record.read();
  // or use pipe
  const fileStream = fs.createWriteStream(`ts_${record.size}.txt`);
  record.pipe(fileStream);
}

Additional context

Bucket.list method should be marked as deprecated and removed in v1.0.0 version.

Code of Conduct

  • I agree to follow this project's Code of Conduct
@atimin atimin added the enhancement New feature or request label Jul 28, 2022
@atimin atimin added this to the v0.6.0 milestone Jul 28, 2022
@atimin atimin self-assigned this Jul 28, 2022
atimin added a commit that referenced this issue Jul 29, 2022
* add method Bucket.query

* update docs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant