Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cloud_storage: warnings on topic creation #3322

Closed
jcsp opened this issue Dec 20, 2021 · 2 comments · Fixed by #3445
Closed

cloud_storage: warnings on topic creation #3322

jcsp opened this issue Dec 20, 2021 · 2 comments · Fixed by #3445
Assignees
Labels
area/cloud-storage Shadow indexing subsystem kind/tech-debt

Comments

@jcsp
Copy link
Contributor

jcsp commented Dec 20, 2021

When creating a topic with shadow indexing enabled, many warning-severity messages are sent to the log (successful creation should not output any warnings).

Some of these are the same messages as during leadership changes (#3310)

WARN  2021-12-20 10:51:59,779 [shard 14] archival - [fiber1 kafka/verifier-1k/33] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:00,817 [shard 5] archival - [fiber1 kafka/verifier-1k/47] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:03,361 [shard 15] archival - [fiber1 kafka/verifier-1k/260] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:03,407 [shard 14] archival - [fiber1 kafka/verifier-1k/162] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:09,362 [shard 9] cloud_storage - [fiber1~12~0|1|9940ms] - remote.cc:224 - Downloading manifest from {violet-desert-kangaroo-vectorized-archival} {key_not_found}, manifest at {"00000000/meta/kafka/verifier-1k/120_40/manifest.json"} not available
WARN  2021-12-20 10:52:09,380 [shard 0] cloud_storage - [fiber1~12~0|1|9960ms] - remote.cc:224 - Downloading manifest from {violet-desert-kangaroo-vectorized-archival} {key_not_found}, manifest at {"20000000/meta/kafka/verifier-1k/262_40/manifest.json"} not available
WARN  2021-12-20 10:52:09,411 [shard 8] cloud_storage - [fiber1~17~0|1|9950ms] - remote.cc:224 - Downloading manifest from {violet-desert-kangaroo-vectorized-archival} {key_not_found}, manifest at {"a0000000/meta/kafka/verifier-1k/182_40/manifest.json"} not available
WARN  2021-12-20 10:52:11,545 [shard 13] archival - [fiber1 kafka/verifier-1k/415] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:12,899 [shard 8] archival - [fiber1 kafka/verifier-1k/182] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:12,918 [shard 10] archival - [fiber1 kafka/verifier-1k/346] - ntp_archiver_service.cc:356 - archival metadata STM update failed
WARN  2021-12-20 10:52:18,625 [shard 6] cloud_storage - [fiber1~10~0|1|9950ms] - remote.cc:224 - Downloading manifest from {violet-desert-kangaroo-vectorized-archival} {key_not_found}, manifest at {"a0000000/meta/kafka/verifier-1k/81_40/manifest.json"} not available
WARN  2021-12-20 10:52:18,629 [shard 3] cloud_storage - [fiber1~10~0|1|9950ms] - remote.cc:224 - Downloading manifest from {violet-desert-kangaroo-vectorized-archival} {key_not_found}, manifest at {"80000000/meta/kafka/verifier-1k/107_40/manifest.json"} not available

@jcsp jcsp added kind/tech-debt area/cloud-storage Shadow indexing subsystem labels Dec 20, 2021
@LenaAn
Copy link
Contributor

LenaAn commented Dec 30, 2021

Logs from archival are fixed after #3371, logs from cloud_storage appear because cloud_storage starts before manifest is available in S3. One idea is to set a timeout before remote starts requesting manifest, but I'm not sure it's worth it

@jcsp
Copy link
Contributor Author

jcsp commented Jan 4, 2022

For the manifest download warning, what about suppressing the message when the topic's high water mark is at zero (i.e. freshly created)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/cloud-storage Shadow indexing subsystem kind/tech-debt
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants