Segments/Tables getting dropped unwantedly

[Shailesh-Pandey]

druid version:0.13.0
<<<<<
dsql> select * from tablename;

java.lang.RuntimeException: Error while applying rule DruidTableScanRule, args [rel#16916567:LogicalTableScan.NONE.[](table=[druid, tablename])]

On the druid overlord console the dropped segments are not visible but a "\d" on dsql shows the table names.

The data ingestion on one of the dropped tables was almost secondly(lot of data) while on another one was just hourly(roughly). All indexes are realtime.
Any idea on what could be the issue or what should be initial approach towards troubleshooting.

Thanks in advance.

[gianm]

@Shailesh-Pandey do you see a full stack trace somewhere, perhaps in the broker logs?

[Shailesh-Pandey]

@Shailesh-Pandey do you see a full stack trace somewhere, perhaps in the broker logs?
From where can I access the broker logs? is it the var folder that get's created for saving segment data?

[Shailesh-Pandey]

I looked up in the var folder and found the logs in the folder known as indexing-logs:
<<<<<<
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/used","value":238120536,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"heap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/init","value":1056964608,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"heap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/max","value":-1,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"nonheap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/committed","value":102432768,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"nonheap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/used","value":100110464,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"nonheap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/mem/init","value":2555904,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"memKind":"nonheap"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/pool/max","value":251658240,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"poolKind":"nonheap","poolName":"Code Cache"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/pool/committed","value":32964608,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"poolKind":"nonheap","poolName":"Code Cache"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/pool/used","value":32495488,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"poolKind":"nonheap","poolName":"Code Cache"}]
2019-03-24T10:52:19,379 INFO [MonitorScheduler-0] org.apache.druid.java.util.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2019-03-24T10:52:19.379Z","service":"druid/middleManager","host":"localhost:8110","version":"0.13.0-incubating","metric":"jvm/pool/init","value":2555904,"dataSource":["table1"],"id":["index_realtime_table1_2019-01-01T00:00:00.000Z_0_0"],"poolKind":"nonheap","poolName":"Code Ca

The last log ended with Ca only
Note: Metadata storage is in derby

This is for one table while the other one does not have any logs.

[ajohnsonz]

@Shailesh-Pandey Did you find a cause/fix for this?

[Shailesh-Pandey]

@Shailesh-Pandey Did you find a cause/fix for this?

No I did not find any fix for it.
"Metadata storage is in derby". I am guessing this might be the cause even though I was not using it for production.

[stale]

This issue has been marked as stale due to 280 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@druid.apache.org list. Thank you for your contributions.

[stale]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.