Skip to content

Regeneration of DISABLE operations when setting instance operations #3045

@LZD-PratyushBhatt

Description

@LZD-PratyushBhatt

Describe the bug

There is a bug in the instance operation handling logic that causes an infinite cycle of regenerated DISABLE operations when applying operations like EVACUATE to an instance with HELIX_ENABLED=false in legacy fields. This prevents proper instance lifecycle transitions and causes unexpected behavior in instance swapping scenarios.

To Reproduce

  1. Create an instance in the cluster.
  2. Set DISABLE operation with USER source:
    Apply to InstanceConf...
    curl -X POST "http://<host>:<port>/admin/v2/clusters/<cluster>/instances/<instance>?command=setInstanceOperation&instanceOperationSource=USER&instanceOperation=DISABLE"
  3. Set EVACUATE operation with AUTOMATION source:
    Apply to InstanceConf...
    curl -X POST "http://<host>:<port>/admin/v2/clusters/<cluster>/instances/<instance>?command=setInstanceOperation&instanceOperationSource=AUTOMATION&instanceOperation=EVACUATE"
  4. Set EVACUATE operation again with AUTOMATION source
  5. Check the instance configuration in ZooKeeper

After the second EVACUATE operation, a new DISABLE operation is generated with:

  1. Updated timestamp (more recent than the EVACUATE timestamp)
  2. Added LEGACY_DISABLED_TYPE field
  3. The cycle repeats with each subsequent evacuate operation

Expected behavior

The DISABLE operation should not be regenerated with new timestamps when setting a different operation. Instance operations should be updated consistently without creating an infinite cycle.

Additional context

RCA:
The regeneration happens because:

  1. getActiveInstanceOperation() returns the last operation (EVACUATE)
  2. EVACUATE is in INSTANCE_DISABLED_OVERRIDABLE_OPERATIONS
  3. This causes getInstanceOperation() to override it with a newly generated DISABLE operation

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions