ttl clean data error

[piaoluoxiaobai]

Please answer these questions before submitting your issue.

• Why do you submit this issue?
• Question or discussion
• Bug
• Requirement
• Feature or performance improvement

---

Question

• What do you want to know?
  how to solve clean historical data error

---

Bug

• Which version of SkyWalking, OS and JRE?
  6.4.0 os:linux jre 1.8

• Which company or project?

• What happen?
  If possible, provide a way for reproducing the error. e.g. demo application, component version.
  i use docker-compose deploy this, like this:

#skywalking的存储
  elasticsearch:
    image: elasticsearch:6.8.1
    container_name: elasticsearch
    restart: always
    ports:
      - 9200
      - 9300
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms4096m -Xmx4096m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
  
  oap:
    image: skywalking-oap-tz:6.4.0
    container_name: oap
    depends_on:
      - elasticsearch
    restart: always
    ports:
      - 11800
      - 12800
    environment:
      SW_STORAGE: elasticsearch
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200
      
      #采样率 The sample rate precision is 1/10000. 10000 means 100% sample in default.
      SW_TRACE_SAMPLE_RATE: 0
      #每1000个请求执行一次
      SW_STORAGE_ES_BULK_ACTIONS: 2000
      #每40mb刷新一次内存块
      SW_STORAGE_ES_BULK_SIZE: 40
      #无论请求数量如何，每30s刷新一次堆
      SW_STORAGE_ES_FLUSH_INTERVAL: 30
      #并发请求的数量
      SW_STORAGE_ES_CONCURRENT_REQUESTS: 4
     
      #设置ttl删除日志 (月)
      SW_CORE_MONTH_METRIC_DATA_TTL: 0
      SW_STORAGE_ES_MONTH_METRIC_DATA_TTL: 0
      #天
      SW_CORE_DAY_METRIC_DATA_TTL: 0
      SW_STORAGE_ES_OTHER_METRIC_DATA_TTL: 0
      SW_STORAGE_ES_RECORD_DATA_TTL: 1
      #小时
      SW_CORE_HOUR_METRIC_DATA_TTL: 24

everything goes well，but when clean historical data，it can't work well with some error:

2020-01-07 12:05:09,383 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer -79208718 [pool-13-thread-1] INFO  [] - Beginning to remove expired metrics from the storage.


2020-01-07 12:05:09,398 - org.elasticsearch.client.RestClient -79208733 [I/O dispatcher 1] WARN  [] - request [PUT http://elasticsearch:9200/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s] returned 1 warnings: [299 Elasticsearch-6.8.1-1fad4e1 "[types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions."]


2020-01-07 12:05:09,399 - org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer -79208734 [pool-13-thread-1] ERROR [] - Remove data in background failure.


org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=resource_already_exists_exception, reason=index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists]


	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.3.2.jar:6.3.2]


	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:653) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:628) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:535) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


	at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:99) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


	at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.createIndex(ElasticSearchClient.java:168) ~[library-client-6.4.0.jar:6.4.0]


	at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.HistoryDeleteEsDAO.deleteHistory(HistoryDeleteEsDAO.java:75) ~[storage-elasticsearch-plugin-6.4.0.jar:6.4.0]


	at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.execute(DataTTLKeeperTimer.java:73) ~[server-core-6.4.0.jar:6.4.0]


	at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.lambda$delete$1(DataTTLKeeperTimer.java:66) ~[server-core-6.4.0.jar:6.4.0]


	at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_181]


	at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.delete(DataTTLKeeperTimer.java:64) ~[server-core-6.4.0.jar:6.4.0]


	at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.4.0.jar:6.4.0]


	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]


	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]


	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]


	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181]


	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]


	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]


	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]


	Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch:9200], URI [/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]


{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"}],"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"},"status":400}


		at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:705) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:198) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:522) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:508) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.IndicesClient.create(IndicesClient.java:99) ~[elasticsearch-rest-high-level-client-6.3.2.jar:6.3.2]


		at org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient.createIndex(ElasticSearchClient.java:168) ~[library-client-6.4.0.jar:6.4.0]


		at org.apache.skywalking.oap.server.storage.plugin.elasticsearch.base.HistoryDeleteEsDAO.deleteHistory(HistoryDeleteEsDAO.java:75) ~[storage-elasticsearch-plugin-6.4.0.jar:6.4.0]


		at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.execute(DataTTLKeeperTimer.java:73) ~[server-core-6.4.0.jar:6.4.0]


		at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.lambda$delete$1(DataTTLKeeperTimer.java:66) ~[server-core-6.4.0.jar:6.4.0]


		at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_181]


		at org.apache.skywalking.oap.server.core.storage.ttl.DataTTLKeeperTimer.delete(DataTTLKeeperTimer.java:64) ~[server-core-6.4.0.jar:6.4.0]


		at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36) [apm-util-6.4.0.jar:6.4.0]


		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]


		at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_181]


		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]


		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_181]


		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]


		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]


		at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]


	Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://elasticsearch:9200], URI [/endpoint_relation_server_side_month-202001?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]


{"error":{"root_cause":[{"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"}],"type":"resource_already_exists_exception","reason":"index [endpoint_relation_server_side_month-202001/0d0aytdDRCm8jvpejAf-WA] already exists","index_uuid":"0d0aytdDRCm8jvpejAf-WA","index":"endpoint_relation_server_side_month-202001"},"status":400}


		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:377) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]


		at org.elasticsearch.client.RestClient$1.completed(RestClient.java:366) ~[elasticsearch-rest-client-6.3.2.jar:6.3.2]


		at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119) ~[httpcore-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177) ~[httpasyncclient-4.1.2.jar:4.1.2]


		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81) ~[httpasyncclient-4.1.2.jar:4.1.2]


		at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39) ~[httpasyncclient-4.1.2.jar:4.1.2]


		at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[httpcore-nio-4.4.5.jar:4.4.5]


		at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[httpcore-nio-4.4.5.jar:4.4.5]

�
		... 1 more


2020-01-07 12:05:12,840 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -79212175 [pool-12-thread-1] INFO  [] - Synchronous bulk took time: 2039 millis, size: 1906


2020-01-07 12:05:18,179 - org.apache.skywalking.oap.server.library.client.elasticsearch.ElasticSearchClient -79217514 [pool-12-thread-1] INFO  [] - Synchronous bulk took time: 2030 millis, size: 1577

Requirement or improvement

• Please describe about your requirements or improvement suggestions.
  i am sure elasticsearch is healthy, how can i solve it

[wu-sheng]

New to me. @dmsolr Any idea what is meaning of this error?

[kezhenxu94]

@piaoluoxiaobai I'm just wondering why you set SW_CORE_MONTH_METRIC_DATA_TTL and SW_STORAGE_ES_MONTH_METRIC_DATA_TTL to 0, this will make the timer to always delete all the month-metrics-related indexes and if you have running agents, they will also generate metrics, which makes the backend to create the indices (if not exist), and the timer also creates the indices, they may both create the same index in a short time, causing resource_already_exists_exception

[piaoluoxiaobai]

@kezhenxu94 I don't want to keep the data too long, just for one day

[wu-sheng]

For one day, TTL = 2

[piaoluoxiaobai]

@wu-sheng hello，it means set SW_CORE_MONTH_METRIC_DATA_TTL to 2?

[piaoluoxiaobai]

or means set SW_STORAGE_ES_OTHER_METRIC_DATA_TTL should more than 2

[wu-sheng]

Any TTL should be greater or equal 2.