Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rollover fails to start on es cluster which was using rollover before #757

Closed
pavolloffay opened this issue Nov 12, 2019 · 1 comment · Fixed by jaegertracing/jaeger#1964
Labels
Elasticsearch The issues related to Elasticsearch storage

Comments

@pavolloffay
Copy link
Member

/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py:847: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  warnings.warn((
Detected ElasticSearch Version 7
Creating index template jaeger-span
{"acknowledged":true}
Creating index jaeger-span-000001
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 559, in do_action
    self.client.indices.create(index=self.name, body=self.body)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/client/indices.py", line 105, in create
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/transport.py", line 343, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 257, in perform_request
    self._raise_error(response.status, raw_data)
  File "/usr/local/lib/python3.8/site-packages/elasticsearch/connection/base.py", line 181, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(
elasticsearch.exceptions.RequestError: RequestError(400, 'resource_already_exists_exception', 'index [jaeger-span-000001/bUHw2FzARlityt-oiTwsCA] already exists')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/es-rollover/esRollover.py", line 234, in <module>
    main()
  File "/es-rollover/esRollover.py", line 64, in main
    perform_action(action, client, write_alias, read_alias, prefix+'jaeger-span', 'jaeger-span')
  File "/es-rollover/esRollover.py", line 82, in perform_action
    create_index(client, index)
  File "/es-rollover/esRollover.py", line 116, in create_index
    raise e
  File "/es-rollover/esRollover.py", line 113, in create_index
    create.do_action()
  File "/usr/local/lib/python3.8/site-packages/curator/actions.py", line 566, in do_action
    raise exceptions.FailedExecution('Index %s already exists.' % self.name)
curator.exceptions.FailedExecution: Index jaeger-span-000001 already exists.

Example CR

# setup an elasticsearch with `make es`
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  query:
    image: "pavolloffay/jaeger-query:es-scripts-tls2"
  collector:
    image: "pavolloffay/jaeger-collector:es-scripts-tls2"
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: https://quickstart-es-http.default.svc:9200
        use-aliases: true
#        tls.ca: /es/secrets/ca.crt
        tls.skip-host-verify: true
#        username: elastic
#        password: ql7hbmqfzzkrtn6klcdsh8n5
    secretName: jaeger-secret
    esIndexCleaner:
      image: "pavolloffay/jaeger-es-index-cleaner:es-scripts-tls2"
      schedule: "*/1 * * * *"
    esRollover:
      image: "pavolloffay/jaeger-es-rollover:es-scripts-tls2"
      schedule: "*/2 * * * *"
  volumeMounts:
    - name: secrets
      mountPath: /es/secrets/
      readOnly: true
  volumes:
    - name: secrets
      secret:
        secretName: quickstart-es-http-certs-public
@pavolloffay pavolloffay added the Elasticsearch The issues related to Elasticsearch storage label Nov 12, 2019
@pavolloffay
Copy link
Member Author

pavolloffay commented Nov 12, 2019

make es

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: simple-prod
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        # Note: This assumes elasticsearch is running in the "default" namespace.
        server-urls: http://elasticsearch.default.svc:9200
        use-aliases: true
    esRollover:
      conditions: '{"max_age": "10s"}'
      schedule: "*/2 * * * *"
      image: pavolloffay/jaeger-es-rollover:rollover-idempotent

and delete the instance and create again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Elasticsearch The issues related to Elasticsearch storage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant