sessions_hourly_local table is empty and error on Releases page

[oymironov]

Environment

self-hosted (https://develop.sentry.dev/self-hosted/)

Version

22.10.0

Link

No response

DSN

No response

Steps to Reproduce

Hi. I found that there are no entries in table sessions_hourly_local.
I found it because of error on the Releases page in sentry web-ui.

snuba-api logs in this moment told that this SQL not working:

SELECT (toStartOfHour(started, 'Universal') AS _snuba_bucketed_started), (project_id AS _snuba_project_id), _snuba_bucketed_started, (plus(countIfMerge(sessions), sumIfMerge(sessions_preaggr)) AS _snuba_sessions), _snuba_project_id FROM sessions_hourly_local PREWHERE in(_snuba_project_id, [12, 9, 11, 17, 10, 8, 5, 6, 15, 27, 7, 28, 26, 13, 168, 16, 14, 270, 29]) WHERE greaterOrEquals((started AS _snuba_started), toDateTime('2023-02-21T14:00:00', 'Universal')) AND less(_snuba_started, toDateTime('2023-02-22T13:33:00', 'Universal')) AND equals((org_id AS _snuba_org_id), 3) AND in(tuple(_snuba_project_id), (tuple(28), tuple(12), tuple(15), tuple(8), tuple(14), tuple(27), tuple(7), tuple(26), tuple(16), tuple(9))) AND in(_snuba_project_id, (7, 8, 9, 12, 14, 15, 16, 26, 27, 28)) GROUP BY _snuba_bucketed_started, _snuba_project_id ORDER BY _snuba_sessions DESC LIMIT 5000 OFFSET 0


DB::Exception: Types of column 1 in section IN don't match: UInt64 on the left, Tuple(UInt64) on the right

and then I found that sessions_hourly_local is empty. We have been working for 3 months!

Expected Result

Releases page works without errors.

Actual Result

snuba-api-log:

2023-02-22 13:32:34,868 Error running query: SELECT (toStartOfHour(started, 'Universal') AS _snuba_bucketed_started), (project_id AS _snuba_project_id), _snuba_bucketed_started, (plus(countIfMerge(sessions), sumIfMerge(sessions_preaggr)) AS _snuba_sessions), _snuba_project_id FROM sessions_hourly_local PREWHERE in(_snuba_project_id, [12, 9, 11, 17, 10, 8, 5, 6, 15, 27, 7, 28, 26, 13, 168, 16, 14, 270, 29]) WHERE greaterOrEquals((started AS _snuba_started), toDateTime('2023-02-21T14:00:00', 'Universal')) AND less(_snuba_started, toDateTime('2023-02-22T13:33:00', 'Universal')) AND equals((org_id AS _snuba_org_id), 3) AND in(tuple(_snuba_project_id), (tuple(28), tuple(12), tuple(15), tuple(8), tuple(14), tuple(27), tuple(7), tuple(26), tuple(16), tuple(9))) AND in(_snuba_project_id, (7, 8, 9, 12, 14, 15, 16, 26, 27, 28)) GROUP BY _snuba_bucketed_started, _snuba_project_id ORDER BY _snuba_sessions DESC LIMIT 5000 OFFSET 0
DB::Exception: Types of column 1 in section IN don't match: UInt64 on the left, Tuple(UInt64) on the right. Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xa3fb75a in /usr/bin/clickhouse
1. DB::Set::checkTypesEqual(unsigned long, std::__1::shared_ptr<DB::IDataType const> const&) const @ 0x14fbe300 in /usr/bin/clickhouse
2. DB::KeyCondition::tryPrepareSetIndex(DB::KeyCondition::FunctionTree const&, std::__1::shared_ptr<DB::Context const>, DB::KeyCondition::RPNElement&, unsigned long&) @ 0x15825144 in /usr/bin/clickhouse
3. DB::KeyCondition::tryParseAtomFromAST(DB::KeyCondition::Tree const&, std::__1::shared_ptr<DB::Context const>, DB::Block&, DB::KeyCondition::RPNElement&) @ 0x15821687 in /usr/bin/clickhouse
4. DB::KeyCondition::traverseAST(DB::KeyCondition::Tree const&, std::__1::shared_ptr<DB::Context const>, DB::Block&) @ 0x1581f3c6 in /usr/bin/clickhouse
5. DB::KeyCondition::traverseAST(DB::KeyCondition::Tree const&, std::__1::shared_ptr<DB::Context const>, DB::Block&) @ 0x1581f5c5 in /usr/bin/clickhouse
6. DB::KeyCondition::traverseAST(DB::KeyCondition::Tree const&, std::__1::shared_ptr<DB::Context const>, DB::Block&) @ 0x1581f5c5 in /usr/bin/clickhouse
7. DB::KeyCondition::traverseAST(DB::KeyCondition::Tree const&, std::__1::shared_ptr<DB::Context const>, DB::Block&) @ 0x1581f5c5 in /usr/bin/clickhouse
8. DB::KeyCondition::KeyCondition(DB::ActionDAGNodes, std::__1::shared_ptr<DB::TreeRewriterResult const>, std::__1::shared_ptr<DB::PreparedSets>, std::__1::shared_ptr<DB::Context const>, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::shared_ptr<DB::ExpressionActions> const&, bool, bool) @ 0x1581ff8f in /usr/bin/clickhouse
9. DB::ReadFromMergeTree::selectRangesToRead(std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const> > >, std::__1::shared_ptr<DB::PrewhereInfo> const&, DB::ActionDAGNodes const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::SelectQueryInfo const&, std::__1::shared_ptr<DB::Context const>, unsigned int, std::__1::shared_ptr<std::__1::unordered_map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, long, std::__1::hash<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::equal_to<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const, long> > > >, DB::MergeTreeData const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, bool, Poco::Logger*) @ 0x16047960 in /usr/bin/clickhouse
10. DB::ReadFromMergeTree::selectRangesToRead(std::__1::vector<std::__1::shared_ptr<DB::IMergeTreeDataPart const>, std::__1::allocator<std::__1::shared_ptr<DB::IMergeTreeDataPart const> > >) const @ 0x16046dd0 in /usr/bin/clickhouse
11. DB::ReadFromMergeTree::getAnalysisResult() const @ 0x16049abd in /usr/bin/clickhouse
12. DB::ReadFromMergeTree::initializePipeline(DB::QueryPipelineBuilder&, DB::BuildQueryPipelineSettings const&) @ 0x1604a00c in /usr/bin/clickhouse
13. DB::ISourceStep::updatePipeline(std::__1::vector<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder> >, std::__1::allocator<std::__1::unique_ptr<DB::QueryPipelineBuilder, std::__1::default_delete<DB::QueryPipelineBuilder> > > >, DB::BuildQueryPipelineSettings const&) @ 0x160130e6 in /usr/bin/clickhouse
14. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x1602d93c in /usr/bin/clickhouse
15. DB::InterpreterSelectWithUnionQuery::execute() @ 0x14ea5e62 in /usr/bin/clickhouse
16. ? @ 0x15197429 in /usr/bin/clickhouse
17. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x15194957 in /usr/bin/clickhouse
18. DB::TCPHandler::runImpl() @ 0x15c8dd6c in /usr/bin/clickhouse
19. DB::TCPHandler::run() @ 0x15ca2379 in /usr/bin/clickhouse
20. Poco::Net::TCPServerConnection::start() @ 0x18a16b73 in /usr/bin/clickhouse
21. Poco::Net::TCPServerDispatcher::run() @ 0x18a17ecd in /usr/bin/clickhouse
22. Poco::PooledThread::run() @ 0x18bda829 in /usr/bin/clickhouse
23. Poco::ThreadImpl::runnableEntry(void*) @ 0x18bd80e2 in /usr/bin/clickhouse
24. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
25. __clone @ 0x12161f in /lib/x86_64-linux-gnu/libc-2.27.so
Traceback (most recent call last):
  File "/usr/src/snuba/./snuba/clickhouse/native.py", line 192, in execute
    result_data = query_execute()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 304, in execute
    rv = self.process_ordinary_query(
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 491, in process_ordinary_query
    return self.receive_result(with_column_types=with_column_types,
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 151, in receive_result
    return result.get_result()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/result.py", line 50, in get_result
    for packet in self.packet_generator:
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 167, in packet_generator
    packet = self.receive_packet()
  File "/usr/local/lib/python3.8/site-packages/clickhouse_driver/client.py", line 184, in receive_packet
    raise packet.exception
clickhouse_driver.errors.ServerException: Code: 53.

[getsantry]

Assigning to @getsentry/support for routing, due by Wednesday, February 22nd at 17:00 (yyz). ⏲️

[getsantry]

Routing to @getsentry/open-source for triage, due by Thursday, February 23rd at 5:00 pm (sea). ⏲️

[hubertdeng123]

Going to route this to search and storage since they'll probably have more context on thesessions_hourly_local table.

[getsantry]

Routing to @getsentry/sns for triage, due by Friday, February 24th at 3:06 pm (sfo). ⏲️

[onewland]

Hi @oymironov,

A couple things we'd like to check here. Are there any obvious sessions-related errors coming from the snuba-sessions-consumer container? After a restart, can you see a sessions-consumer loading?

Take a look with:

$ docker-compose logs -f snuba-sessions-consumer

(https://github.com/getsentry/self-hosted/blob/master/docker-compose.yml#L253-L256)

If the snuba-sessions-consumer appears to start but has no errors, it may be that sessions aren't coming into Snuba at all and the ingest level needs to be verified.

I don't have self-hosted installed locally, but on your machine a command like this should let you run commands on the kafka container:

$ docker-compose exec kafka -it bash

Then, when inside the container, something like:

$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic ingest-sessions

Then send your installation some sessions. If there are no messages on that topic, then this is a Relay issue and I think that team would be better for troubleshooting it.

[oymironov]

Hi, @onewland
Logs of snuba-sessions-consumer are not empty, looks like usual consumer work:
2023-02-25 06:15:08,404 Completed processing <Batch: 84 messages, open for 0.82 seconds>.
2023-02-25 06:15:09,151 Completed processing <Batch: 121 messages, open for 0.81 seconds>.
2023-02-25 06:15:09,913 Completed processing <Batch: 105 messages, open for 0.82 seconds>.
2023-02-25 06:15:10,683 Completed processing <Batch: 96 messages, open for 0.83 seconds>.

and topic ingest-sessions in kafka, sure, not empty. A lot of messages there were:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --describe --topic ingest-sessions
Topic: ingest-sessions PartitionCount: 3 ReplicationFactor: 3 Configs: min.insync.replicas=1,segment.bytes=104857600,retention.ms=43200000,message.format.version=2.7-IV2
Topic: ingest-sessions Partition: 0 Leader: 3 Replicas: 3,4,1 Isr: 1,4,3
Topic: ingest-sessions Partition: 1 Leader: 0 Replicas: 0,1,2 Isr: 0,2,1
Topic: ingest-sessions Partition: 2 Leader: 4 Replicas: 4,2,3 Isr: 4,2,3

[getsantry]

Routing to @getsentry/ingest for triage, due by Tuesday, February 28th at 17:00 (vie). ⏲️

[getsantry]

Routing to @getsentry/sns for triage, due by Tuesday, February 28th at 5:00 pm (sfo). ⏲️