Cannot update from a previous version to current snapshot

[ThorbenLindhauer]

Describe the bug

• When updating a system to the current version, the schema validation fails on startup with a problem on the web-session index

To Reproduce

Steps to reproduce the behavior:

1. Create a C8 cluster with Operate
2. Enable persistent sessions (set property camunda.operate.persistentSessionsEnabled to true, or use the corresponding environment variable)
3. Log into Operate, so that a document is written to the web-session index
4. Update Operate to the current snapshot

Alternative: the same should work with a current snapshot and restarting Operate. It doesn't matter if this is a true version update.

Current behavior

• Operate does not start successfully
• Stack trace:

io.camunda.operate.exceptions.OperateRuntimeException: Index name: web-session. Not supported index changes are introduced. Data migration is required. Changes found: [PropertyDifference{name='attributes', leftValue=IndexMappingProperty{name='attributes', typeDefinition={type=object}}, rightValue=IndexMappingProperty{name='attributes', typeDefinition={properties={com={properties={auth0={properties={state={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}}, requestedUrl={type=text, fields={keyword={ignore_above=256, type=keyword}}}, SPRING_SECURITY_CONTEXT={type=text, fields={keyword={ignore_above=256, type=keyword}}}, SPRING_SECURITY_SAVED_REQUEST={type=text, fields={keyword={ignore_above=256, type=keyword}}}}}}}]
	at io.camunda.operate.schema.IndexSchemaValidator.validateDifferenceAndCollectNewFields(IndexSchemaValidator.java:198) ~[operate-schema-8.5.0-SNAPSHOT.jar!/:8.5.0-SNAPSHOT]
	at io.camunda.operate.schema.IndexSchemaValidator.validateIndexMappings(IndexSchemaValidator.java:172) ~[operate-schema-8.5.0-SNAPSHOT.jar!/:8.5.0-SNAPSHOT]
	at io.camunda.operate.schema.SchemaStartup.initializeSchema(SchemaStartup.java:61) ~[operate-schema-8.5.0-SNAPSHOT.jar!/:8.5.0-SNAPSHOT]

Expected behavior

• Operate starts successfully without complaints

Environment

• OS: [e.g. MacOS]
• Browser: [e.g. chrome, safari]
• Operate Version: [e.g. 8.1]

Additional context

This behavior was introduced with https://github.com/camunda/operate/issues/6483

Suspected root causes:

• The web session index defines "dynamic": true in its schema definition. This means that the schema is extended as documents are inserted, depending on which fields they define
• For the attributes property, this usually contains multiple fields that are written to the index and thus change the schema (see the stack trace)
• The index validator does not expect the case of dynamic mappings and is too strict

Solution options:

• as a quick fix, do not validate dynamic indexes. Currently only the web-session index uses that feature.

---

Acceptance Criteria

Definition of Ready - Checklist

• The bug has been reproduced by the assignee in the environment compatible with the provided one; otherwise, the issue is closed with a comment
• The issue has a meaningful title, description, and testable acceptance criteria
• The issue has been labeled with an appropriate Bug-area label
• Necessary screenshots, screen recordings, or files are attached to the bug report

For UI changes required to solve the bug:

• Design input has been collected by the assignee

Implementation

🔍 Root Cause Analysis

💭 Proposed Solution

👉 Handover Dev to QA

• Changed components:
• Side effects on other components:
• Handy resources:
  BPMN/DMN models, plugins, scripts, REST API endpoints + example payload, etc :
• Example projects:

• Commands/Steps needed to test; Versions to validate:

• Docker file / HELM chart : in case that it needed to be tested via docker share the version contain the fixed along with version of other services .

• Release version ( in which version this fixed/feature will be released):

📗 Link to the test case

[ThorbenLindhauer]

Some hints for reproducing this:

• Verify that a session has been persisted: http://localhost:9200/operate-web-session-1.1.0_/_search
• URL to check the web session mappings: http://localhost:9200/operate-web-session-1.1.0_/_mapping (should change after a session is persisted)

Assuming ES is running on localhost:9200

Using the Makefile and Docker Compose, I made these changes:

• make env-up: Add CAMUNDA_OPERATE_PERSISTENT_SESSIONS_ENABLED=true as an environment variable to the Maven command starting Operate
• Docker compose elasticsearch: add a volume ./es-data:/usr/share/elasticsearch/data to persist ES data
• Docker compose zeebe: add a volume ./zeebe-data:/usr/local/zeebe/data to persist Zeebe data

[ThorbenLindhauer]

Here is how I tested the update scenario locally:

I built an Operate Docker image from the current branch by running mvn clean install -DskipTests and then docker build -t thorben/operate:SNAPSHOT -f operate.Dockerfile --target app . from the root directory.

I declared Operate in the Docker compose file as follows:

  operate:
    container_name: operate
    image: thorben/operate:SNAPSHOT
    environment:
      - CAMUNDA_OPERATE_TASKLIST_URL=http://localhost:8081
      - CAMUNDA_OPERATE_PERSISTENT_SESSIONS_ENABLED=true
      - CAMUNDA_OPERATE_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_OPERATE_ZEEBE_ELASTICSEARCH_URL=http://elasticsearch:9200
      - CAMUNDA_OPERATE_ZEEBE_GATEWAY_ADDRESS=zeebe:26500
    ports:
      - 8080:8080
    restart: always

I added a volume declaration to the Zeebe and Elasticsearch containers as described in my previous comment.

Then:

1. Run docker-compose -f docker-compose.yml up -d elasticsearch zeebe operate
2. Log into the webapps (important to create a persistent session), deploy a process model, start a process instance
3. Stop the stack
4. Change the versions of the Zeebe and Operate images
5. Run docker compose again