Skip to content

update cloud env #1057

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fmassot merged 2 commits into from
Jan 11, 2022
Merged

update cloud env #1057

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f90297
update cloud env
PSeitz Jan 10, 2022
bdde491
Merge branch 'main' into cloud_docs
fmassot Jan 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 25 additions & 10 deletions docs/administration/cloud-env.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,40 @@
---
title: Operating in the cloud
title: Cloud operation costs
sidebar_position: 1
---

Quickwit has only been tested on AWS S3. This page sums up what we have learned from that.
Quickwit has been tested on Amazon S3. This page sums up what we have learned from that experience.

## Data transfers costs and latency

Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency. For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network.
Cloud providers charge for data transfers in and out of their networks. In addition, querying an index from a remote machine adds some extra latency.
For those reasons, we recommend that you test and use the Quickwit from an instance located within your cloud provider's network.

## Optimizing bandwidth with wisely chosen instances

To get the best performance out of Quickwit search from object storage, we recommend picking an instance with high network bandwidth.
In our experience, `c5n.2xlarge` instances offer the bigger bang for your buck.
We recommend picking instances with high network performance to allow faster downloads from Amazon S3. In our experience, `c5n.2xlarge` instances offer the best bang for your buck.

## GET/PUT requests costs
## Requests cost

A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3, so you don't need to worry too much about it.
A final note on object storage requests costs. These are [quite low](https://aws.amazon.com/s3/pricing/) actually, $0,0004 / 1000 requests for GET and $0.005 / 1000 requests for PUT on AWS S3.

Indeed when indexing, Quickwit generates [splits](../overview/architecture.md#splits) of 5 millions documents each and then
upload them. As they are composed of 9 files, this generates 9 PUT requests per split. When querying one term, Quickwit only needs to make 3 GET requests per split.
### PUT requests

Of course, these requests could add up quickly if you have a large amount of requests.
During indexing, Quickwit uploads new splits on Amazon S3 and progressively merges them until they reach 10 million documents that we call “mature splits”. Such splits have a typical size between 1GB and 10GB and will usually require 2 PUT requests to be uploaded (1 PUT request / 5GB).

With default indexing parameters `commit_timeout_secs` of 60 seconds and `merge_policy.merge_factor` of 10 and assuming you want to ingest 1 million documents every minute, this will cost you less than $1 / month.

### GET requests

When querying, Quickwit needs to make multiple GET requests:

```jsx
#num requests = #num splits * ((#num search fields * #num terms * 3) + #num fast fields)
Copy link
Collaborator

@fmassot fmassot Jan 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to not put fast field number as today I think we are only getting the timestamp fast field values, right? So I would put only 1.

Copy link
Collaborator Author

@PSeitz PSeitz Jan 11, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes you are right, timestamp and sort_by, but sort_by is deacticated


/// Extracts all fast field names.
fn extract_fast_field_names(doc_mapper: &dyn DocMapper) -> HashSet<String> {
    let mut fast_fields = HashSet::new();
    if let Some(timestamp_field) = doc_mapper.timestamp_field_name() {
        fast_fields.insert(timestamp_field);
    }
    if let SortBy::FastField { field_name, .. } = doc_mapper.sort_by() {
        fast_fields.insert(field_name);
    }
    fast_fields
}

```

The above formula assumes that the hotcache is cached, which will be loaded after the first query for every split.

When positions are not enabled, only 2 GET requests will be executed per term.

These requests costs could add up quickly if you have a high number of splits or QPS > 10.
Don't hesitate to [contact us](mailto:hello@quickwit.io) if this is the case :).