Google Cloud Storage error on unconfigured region

[tylerkovacs]

I have followed the steps to configure GCS storage:
https://quickwit.io/docs/guides/storage-setup/gcs-setup

However, when the indexer attempts writes to GCS, it returns the following error:

2023-07-06T19:07:40.971Z  WARN index-doc-batches{index_id=redacted source_id=_ingest-api-source pipeline_ord=0 workbench_id=01H4P9MBAHWYH30G8PWPHPAE4B}:uploader: quickwit_indexing::actors::uploader: Failed to upload split. Killing! cause=Failed uploading key 01H4P9MBB0CH9DGCD3X0N3DYG3.split in bucket s3://redacted/quickwit/production/indexes/redacted
Caused by:
    0: StorageError(kind=InternalError, source=failed to construct request: A region must be set when sending requests to S3. (ConstructionFailure(ConstructionFailure { source: ResolveEndpointError { message: "A region must be set when sending requests to S3.", source: None } })))
    1: failed to construct request: A region must be set when sending requests to S3. (ConstructionFailure(ConstructionFailure { source: ResolveEndpointError { message: "A region must be set when sending requests to S3.", source: None } })) split_id="01H4P9MBB0CH9DGCD3X0N3DYG3"

I was able to resolve this issue by setting the AWS region via the AWS_REGION environment variable. Please update the GCS setup docs to include region configuration if it is truly required.

Thank you, quickwit is very interesting and I'm eager to evaluate it further.

[guilload]

Hi @tylerkovacs,

Yes, the documentation needs to be updated. The next minor release of Quickwit (0.6.2) in a few weeks will address this issue. Then, the best way to configure Quickwit for GCS will be:

Passing credentials via env vars and using the following storage settings in the config file:

storage:
  s3:
    flavor: gcs
    region: us-east1
    endpoint: https://storage.googleapis.com

[fmassot]

@@tylerkovacs, we have just released the 0.6.2. Can you confirm it solves your problem?

[tylerkovacs]

Thanks @fmassot , I'm on holiday this week but will confirm first thing when I return next week.

[tylerkovacs]

@fmassot I have upgraded to Quickwit 0.6.2 with the suggested config and have confirmed that it is working as expected. Thank you!

[tylerkovacs]

That said, it appears that the documentation has not yet been updated to mention the new flavor-based configuration:
https://quickwit.io/docs/reference/storage-uri#google-cloud-storage

[trinity-1686a]

flavor are mentioned in the node-configuration page. The storage-uri page could probably be simplified now, and should link back to the node-configuration page to detail how flavors work

[tylerkovacs]

Thanks @trinity-1686a. This is slightly off-topic but I tried to find a reference to flavor in your docs before writing that last comment using this Google search. There are still no results for that query.

This is a little odd since Google normally reindexes content changes very quickly. My guess is that this is due to your sitemap only including links to docs for an old version (0.5.2.). I noticed that your site is missing a robots.txt file which may also be contributing to the issue.

[fmassot]

thanks @tylerkovacs! I will fix the sitemap indeed.

You can still use the search bar on quickwit.io like in the screenshot.

[guilload]

I've updated the documentation and deployed it.