Skip to content

(2026.06) Snapshot apply fails when SSO is enabled on the source environment

Junbo Liang edited this page Jun 30, 2026 · 1 revision

Bug description

When applying a snapshot to a RES 2026.06 environment, the apply process fails if SSO was enabled on the source environment. The snapshot contains the sso_enabled=True flag but does not include the required OIDC/SAML configuration parameters (e.g., oidc_client_id), causing the SSO auto-configuration step to fail with an error such as SSO auto-configuration failed: [GENERAL_ERROR] oidc_client_id is required.

Affected versions

2026.06

Mitigation

  1. Create an S3 bucket with ACLs disabled.
  2. Download patch_host.py and idea-cluster-manager-2026.06-ed153413.tar.gz by replacing <output-directory> with the directory to download the patch script, <environment-name> with the name of your RES environment, and <bucket-name> with the name of an ACLs disabled S3 bucket under the account/region where RES is deployed, and <partition> with Classic or GovCloud:
    1. The patch applies to RES 2026.06
    2. The patch script requires AWS CLI v2, Python 3.9.16 or above, and Boto3.
    3. Configure the AWS CLI for the account / region where RES is deployed, and make sure that you have S3 permissions to write to the bucket provided through <bucket-name>.
OUTPUT_DIRECTORY=<output-directory>
ENVIRONMENT_NAME=<environment-name>
BUCKET_NAME=<bucket-name>
PARTITION=<partition>

mkdir -p ${OUTPUT_DIRECTORY}
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/2026.06/patch_scripts/patch_host.py --output ${OUTPUT_DIRECTORY}/patch_host.py
curl https://research-engineering-studio-us-east-1.s3.us-east-1.amazonaws.com/releases/2026.06/patch_scripts/patches/idea-cluster-manager-2026.06-ed153413.tar.gz --output ${OUTPUT_DIRECTORY}/idea-cluster-manager-2026.06-ed153413.tar.gz
  1. Run the following patch command:
python3 ${OUTPUT_DIRECTORY}/patch_host.py --environment-name ${ENVIRONMENT_NAME} --module cluster-manager --zip-file ${OUTPUT_DIRECTORY}/idea-cluster-manager-2026.06-ed153413.tar.gz --s3-bucket ${BUCKET_NAME} --partition ${PARTITION}
  1. Restart the Cluster Manager instance for your environment. You may also terminate the instance from the Amazon EC2 Management Console.
INSTANCE_ID=$(aws ec2 describe-instances \
            --filters \
            Name=tag:Name,Values=${ENVIRONMENT_NAME}-cluster-manager \
            Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME}\
            Name=instance-state-name,Values=running \
            --query "Reservations[0].Instances[0].InstanceId" \
            --output text)

aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}
  1. Verify the new Cluster Manager instance status by checking the activity of the auto scaling group starting with the name <environment-name>-cluster-manager-asg. Wait until the new instance is launched successfully.
  2. Re-apply the snapshot. The patched cluster-manager will gracefully skip the SSO auto-configuration step and log a warning. After the snapshot is applied successfully, configure SSO manually through the RES web portal.

Rollback

  1. Restore the original package URI in DynamoDB (replace <region> with the region where RES is deployed):
REGION=<region>

aws dynamodb update-item \
   --table-name ${ENVIRONMENT_NAME}.cluster-settings \
   --key '{"key": {"S": "cluster-manager.app_package_uri"}}' \
   --update-expression "SET #val = :val" \
   --expression-attribute-names '{"#val": "value"}' \
   --expression-attribute-values '{":val": {"S": "s3://research-engineering-studio-'${REGION}'/releases/2026.06/idea-cluster-manager-2026.06.tar.gz"}}'
  1. Terminate the Cluster Manager instance to pick up the restored URI:
INSTANCE_ID=$(aws ec2 describe-instances \
            --filters \
            Name=tag:Name,Values=${ENVIRONMENT_NAME}-cluster-manager \
            Name=tag:res:EnvironmentName,Values=${ENVIRONMENT_NAME}\
            Name=instance-state-name,Values=running \
            --query "Reservations[0].Instances[0].InstanceId" \
            --output text)

aws ec2 terminate-instances --instance-ids ${INSTANCE_ID}

Clone this wiki locally