[Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 #171630

kpollich · 2023-11-21T12:16:44Z

Ref https://discuss.elastic.co/t/fleet-initial-error/347225/7

We're seeing various errors related to encrypted saved objects across multiple plugins in 8.11.1 including Fleet and Rules + Alerts.

Examples

During Fleet Setup

Fleet setup throws an error for each agent policy when we try to generate uninstall tokens.

This will result in Fleet Setup failing, Fleet being unavailable, or the uninstall tokens page being unavailable.

2023-11-16T13:35:52.319+00:00][INFO ][plugins.fleet] Beginning fleet setup
[2023-11-16T13:35:55.464+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.465+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.466+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.466+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.467+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.467+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.468+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.468+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.478+00:00][ERROR][plugins.encryptedSavedObjects] Failed to decrypt "token" attribute: Unsupported state or unable to authenticate data
[2023-11-16T13:35:55.479+00:00][ERROR][plugins.fleet] Uninstall Token is missing the token.

This error originates in Fleet setup here:

kibana/x-pack/plugins/fleet/server/services/setup.ts

Lines 167 to 178 in 3a58207

    
           logger.debug('Generating Agent uninstall tokens'); 
        
           if (!appContextService.getEncryptedSavedObjectsSetup()?.canEncrypt) { 
        
             logger.warn( 
        
               'xpack.encryptedSavedObjects.encryptionKey is not configured, agent uninstall tokens are being stored in plain text' 
        
             ); 
        
           } 
        
           await appContextService.getUninstallTokenService()?.generateTokensForAllPolicies(); 
        
           if (appContextService.getEncryptedSavedObjectsSetup()?.canEncrypt) { 
        
             logger.debug('Checking for and encrypting plain text uninstall tokens'); 
        
             await appContextService.getUninstallTokenService()?.encryptTokens(); 
        
           }

Specifically, the final error is thrown in this function:

kibana/x-pack/plugins/fleet/server/services/security/uninstall_token_service/index.ts

Lines 499 to 503 in 3a58207

    
           private assertToken(attributes: UninstallTokenSOAttributes | undefined) { 
        
             if (!attributes?.token && !attributes?.token_plain) { 
        
               throw new UninstallTokenError('Uninstall Token is missing the token.'); 
        
             } 
        
           }

The encrypted saved objects error is thrown here:

kibana/x-pack/plugins/encrypted_saved_objects/server/crypto/encrypted_saved_objects_service.ts

Lines 545 to 557 in 3a58207

    
             } catch (err) { 
        
               this.options.logger.error( 
        
                 `Failed to decrypt "${attributeName}" attribute: ${err.message || err}` 
        
               ); 
        
               throw new EncryptionError( 
        
                 `Unable to decrypt attribute "${attributeName}"`, 
        
                 attributeName, 
        
                 EncryptionErrorOperation.Decryption, 
        
                 err 
        
               ); 
        
             } 
        
           }

In the Rules and Alerts Plugin

I don't have logs for this, but a discuss user provided this screenshot:

The error is likely identical to the Fleet issue, however the user was able to regenerate their rules' API keys in order to alleviate the problem.

Investigation

We've been able to reproduce a subset of these errors by doing the following

Boot up a self-hosted stack on 8.11.1 with xpack.encryptedSavedObjects.encryptionKey set to some value
Set up Fleet, create an agent policy, and ensure an uninstall token is generated for it
Stop Kibana and change the encryption key value in kibana.yml
Restart Kibana and observe the "failed to decrypt" errors when visiting the uninstall token page

Open questions

Why is this happening on 8.11.1 - is there something special about encrypted saved objects in this patch release?
Should plugins be "re-encrypting" encrypted attributes like API keys, tokens, etc? Why haven't we had to do this before?
Is there a workaround for the Fleet uninstall token issue specifically?

The text was updated successfully, but these errors were encountered:

elasticmachine · 2023-11-21T12:16:47Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

elasticmachine · 2023-11-21T12:16:47Z

Pinging @elastic/fleet (Team:Fleet)

kpollich · 2023-11-21T12:17:01Z

Ping @elastic/kibana-security (Sorry I don't know your team label)

kpollich · 2023-11-21T19:37:27Z

Posting a summary of my findings and what the great folks on the Kibana Security team, ECK team, and Security Defend Workflow teams have helped out with today over Slack.

The "failed to decrypt" errors coming from the encryptedSavedObjects plugin do not seem entirely limited to Fleet
Other "failed to decrypt" errors are present in other plugins, though Fleet's setup lifecycle is called on Kibana startup, so tends to be the loudest and fastest indicator of this issue
Our most clear reports of this bug have come from ECK users
ECK preserves the value of xpack.encryptedSavedObjects.encryptionKey between stack upgrades, so there should not be any kind of key rotation process happening here
The errors in the Fleet plugin are happening during Fleet's setup lifecycle and originate from the getDecryptedTokens method:

kibana/x-pack/plugins/fleet/server/services/security/uninstall_token_service/index.ts

Lines 193 to 228 in be46cce

    
           private getDecryptedTokens = async ( 
        
             options: Partial<SavedObjectsCreatePointInTimeFinderOptions> 
        
           ): Promise<UninstallToken[]> => { 
        
             const tokensFinder = 
        
               await this.esoClient.createPointInTimeFinderDecryptedAsInternalUser<UninstallTokenSOAttributes>( 
        
                 { 
        
                   type: UNINSTALL_TOKENS_SAVED_OBJECT_TYPE, 
        
                   perPage: SO_SEARCH_LIMIT, 
        
                   ...options, 
        
                 } 
        
               ); 
        
             let tokenObject: Array<SavedObjectsFindResult<UninstallTokenSOAttributes>> = []; 
        
             for await (const result of tokensFinder.find()) { 
        
               tokenObject = result.saved_objects; 
        
               break; 
        
             } 
        
             tokensFinder.close(); 
        
             const uninstallTokens: UninstallToken[] = tokenObject.map( 
        
               ({ id: _id, attributes, created_at: createdAt }) => { 
        
                 this.assertPolicyId(attributes); 
        
                 this.assertToken(attributes); 
        
                 this.assertCreatedAt(createdAt); 
        
                 return { 
        
                   id: _id, 
        
                   policy_id: attributes.policy_id, 
        
                   token: attributes.token || attributes.token_plain, 
        
                   created_at: createdAt, 
        
                 }; 
        
               } 
        
             ); 
        
             return uninstallTokens; 
        
           };

I've tried a few different ways to recreate this outside of an ECK environment without success, e.g.

Create a self hosted cluster on 8.9.x or 8.10.x
Set xpack.encryptedSavedObjects.encryptionKey to some value
- https://generate-random.org/encryption-key-generator
Start up Kibana and create an Agent Policy via Fleet
Observe that no uninstall token was created
Upgrade the cluster to 8.11.x
Make sure the same encryptionKey value is used on the upgraded cluster (this is the behavior that ECK employs)
Start Kibana, observe Fleet setup logs
Check the Fleet plugin's new Uninstall Tokens tab and verify that the uninstall token for your policy was generated as expected

I've tried the above steps going from 8.9.2 -> 8.11.1, 8.10.4 -> 8.11.1, and 8.9.2 -> 8.11.1 all without a reproduction of the errors.

Something else to note: the uninstall tokens seem to be the most common culprit for these error logs, but Fleet has another encrypted saved object code path that we haven't seen errors reported for yet:

kibana/x-pack/plugins/fleet/server/services/security/message_signing_service.ts

Lines 55 to 105 in be46cce

    
           public async generateKeyPair( 
        
             providedPassphrase?: string 
        
           ): ReturnType<MessageSigningServiceInterface['generateKeyPair']> { 
        
             const existingKeyPair = await this.checkForExistingKeyPair(); 
        
             if (existingKeyPair) { 
        
               return existingKeyPair; 
        
             } 
        
             const passphrase = providedPassphrase || this.generatePassphrase(); 
        
             const keyPair = generateKeyPairSync('ec', { 
        
               namedCurve: 'prime256v1', 
        
               privateKeyEncoding: { 
        
                 type: 'pkcs8', 
        
                 format: 'der', 
        
                 cipher: 'aes-256-cbc', 
        
                 passphrase, 
        
               }, 
        
               publicKeyEncoding: { 
        
                 type: 'spki', 
        
                 format: 'der', 
        
               }, 
        
             }); 
        
             const privateKey = keyPair.privateKey.toString('base64'); 
        
             const publicKey = keyPair.publicKey.toString('base64'); 
        
             let keypairSoObject: Partial<MessageSigningKeys> = { 
        
               private_key: privateKey, 
        
               public_key: publicKey, 
        
             }; 
        
             keypairSoObject = this.isEncryptionAvailable 
        
               ? { 
        
                   ...keypairSoObject, 
        
                   passphrase, 
        
                 } 
        
               : { ...keypairSoObject, passphrase_plain: passphrase }; 
        
             try { 
        
               await this.soClient.create<Partial<MessageSigningKeys>>( 
        
                 MESSAGE_SIGNING_KEYS_SAVED_OBJECT_TYPE, 
        
                 keypairSoObject 
        
               ); 
        
               return { 
        
                 privateKey, 
        
                 publicKey, 
        
                 passphrase, 
        
               }; 
        
             } catch (error) { 
        
               throw new MessageSigningError(`Error creating key pair: ${error.message}`, error); 
        
             } 
        
           }

If these saved objects are encountering the same decryption errors as the uninstall tokens, we'd expect to see Failed to decrypt "passphrase" attribute: Unsupported state or unable to authenticate data errors during Fleet setup before the uninstall token errors.

Finally, I have reason to suspect this limited to ECK. Kibana does not implicitly set xpack.encryptedSavedObjects.encryptionKey in distributed versions of Kibana. So, unless a self-hosted user has explicitly set this value via kibana.yml and then rotated it improperly (see docs), they wouldn't see any kind of error like this. When this key is unset for distributable versions of Kibana, you'll see logs like this during Kibana startup (I copy/pasted these from my local cluster running production builds of Kibana + ES 8.11.1):

[2023-11-21T14:24:05.431-05:00][WARN ][plugins.encryptedSavedObjects] Saved objects encryption key is not set. This will severely limit Kibana functionality. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.

[2023-11-21T14:24:05.466-05:00][WARN ][plugins.alerting] APIs are disabled because the Encrypted Saved Objects plugin is missing encryption key. Please set xpack.encryptedSavedObjects.encryptionKey in the kibana.yml or use the bin/kibana-encryption-keys command.

# During Fleet setup, we warn on this exact case as well
[2023-11-21T14:24:23.660-05:00][WARN ][plugins.fleet] xpack.encryptedSavedObjects.encryptionKey is not configured, private key passphrase is being stored in plain text
[2023-11-21T14:24:23.665-05:00][WARN ][plugins.fleet] xpack.encryptedSavedObjects.encryptionKey is not configured, agent uninstall tokens are being stored in plain text

So it appears to me that something has potentially changed the value of the encryptionKey config property on ECK clusters in the 8.11.0 or 8.11.1 release. ECK generates a random value for the xpack.encryptedSavedObjects.encryptionKey here: https://github.com/elastic/cloud-on-k8s/blob/main/pkg/controller/kibana/config_settings.go#L251.

Is there any kind of trigger that would cause this random value to change during a stack upgrade? If a K8's namespace or secret or ConfigMap or something was blown away for some reason during the upgrade, would that cause a new encryption key to be randomly generated and break decryption like this? @elastic/cloud-k8s could you perhaps weigh in on this?

kpollich · 2023-11-21T23:10:57Z

I need to debug some local environment stuff with Kubernetes, but I'll try and reproduce the upgrade issue specifically in an ECK setup tomorrow morning my time (eastern US).

kpollich · 2023-11-22T13:25:57Z

I spun up an ECK deployment on my local machine this morning and performed a stack upgrade from 8.9.2 -> 8.11.1 without issue. The value of xpack.encryptedSavedObjects.encryptionKey was consistent across stack versions as expected.

So, my hunch that the ECK operator changed the encryption key seems to not be accurate, at least in this instance. If we can get some kibana logs from before and after the stack upgrade for one of the instances where this bug is happening, that'd be massively helpful in tracking down the cause.

Full findings below:

8.9.2 cluster

# Kibana logs w/ hashed encryption key. Key is generated randomly by ECK as it's not explicitly set in Kibana config
[2023-11-22T12:53:03.208+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: BOKS+JGibEowGQ+xyWsAWQ06rlqQHgFs3OJqaw/THZg=

Created 3 agent policies via Fleet UI

GET kbn:/api/fleet/agent_policies

{
  "items": [
    {
      "id": "04010160-8939-11ee-aa3d-eb15ae9ef4d3",
      "name": "Agent Policy 3",
      "description": "",
      "namespace": "default",
      "monitoring_enabled": [
        "logs",
        "metrics"
      ],
      "inactivity_timeout": 1209600,
      "status": "active",
      "is_managed": false,
      "revision": 1,
      "updated_at": "2023-11-22T13:14:06.844Z",
      "updated_by": "elastic",
      "schema_version": "1.1.1",
      "is_protected": false,
      "agents": 0
    },
    {
      "id": "f845fe40-8936-11ee-aa3d-eb15ae9ef4d3",
      "name": "Agent Policy 2",
      "description": "",
      "namespace": "default",
      "monitoring_enabled": [
        "logs",
        "metrics"
      ],
      "inactivity_timeout": 1209600,
      "status": "active",
      "is_managed": false,
      "revision": 1,
      "updated_at": "2023-11-22T12:59:28.166Z",
      "updated_by": "elastic",
      "schema_version": "1.1.1",
      "is_protected": false,
      "agents": 0
    },
    {
      "id": "efe25190-8936-11ee-aa3d-eb15ae9ef4d3",
      "name": "Agent Policy 1",
      "description": "",
      "namespace": "default",
      "monitoring_enabled": [
        "logs",
        "metrics"
      ],
      "inactivity_timeout": 1209600,
      "status": "active",
      "is_managed": false,
      "revision": 1,
      "updated_at": "2023-11-22T12:59:14.090Z",
      "updated_by": "elastic",
      "schema_version": "1.1.1",
      "is_protected": false,
      "agents": 0
    }
  ],
  "total": 3,
  "page": 1,
  "perPage": 20
}

Uninstall tokens feature is not active, so no tokens exist (expected)

GET kbn:/api/fleet/uninstall_tokens
{
  "statusCode": 404,
  "error": "Not Found",
  "message": "Not Found"
}

GET .kibana_ingest/_search?q=type:fleet-uninstall-tokens
{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

8.11.1 cluster

Everything worked as expected 🙁

[2023-11-22T13:20:25.641+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: BOKS+JGibEowGQ+xyWsAWQ06rlqQHgFs3OJqaw/THZg=

GET kbn:/api/fleet/uninstall_tokens
{
  "items": [
    {
      "id": "f0f540d0-8939-11ee-a3d6-65b5ebccf826",
      "policy_id": "04010160-8939-11ee-aa3d-eb15ae9ef4d3",
      "created_at": "2023-11-22T13:20:44.380Z"
    },
    {
      "id": "f0f540d2-8939-11ee-a3d6-65b5ebccf826",
      "policy_id": "efe25190-8936-11ee-aa3d-eb15ae9ef4d3",
      "created_at": "2023-11-22T13:20:44.380Z"
    },
    {
      "id": "f0f540d1-8939-11ee-a3d6-65b5ebccf826",
      "policy_id": "f845fe40-8936-11ee-aa3d-eb15ae9ef4d3",
      "created_at": "2023-11-22T13:20:44.380Z"
    }
  ],
  "total": 3,
  "page": 1,
  "perPage": 20
}

GET .kibana_ingest/_search?q=type:fleet-uninstall-tokens
{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 5.049856,
    "hits": [
      {
        "_index": ".kibana_ingest_8.9.2_001",
        "_id": "fleet-uninstall-tokens:f0f540d0-8939-11ee-a3d6-65b5ebccf826",
        "_score": 5.049856,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "04010160-8939-11ee-aa3d-eb15ae9ef4d3",
            "token": "bLfvRRZgHaGrCTn1xoPQwe35iFcALvoPTRnJYQ30LVPzwWnSydysabBeYQPCRoNlu+35+0qbkhtH4PdcpzudMoFeHVt++ZO1S0x0Fq7lPyL6TEwJDpeQBiHO+NpAAG67EbMqS8P7doR8CZPfd8GbkLoyCILcEx260IfNAYea"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:20:44.380Z",
          "created_at": "2023-11-22T13:20:44.380Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.9.2_001",
        "_id": "fleet-uninstall-tokens:f0f540d2-8939-11ee-a3d6-65b5ebccf826",
        "_score": 5.049856,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "efe25190-8936-11ee-aa3d-eb15ae9ef4d3",
            "token": "dAPbiKltWkcG5PAmnRWPSLgAEDIi2uMaxy1umTL35Z8qtwKmBDn8/Qu+nIZQett+gPASciI7r75/FQv6XiHgaXezrr89bUbPLzCPd1mWgfA853Ng5UYXuaEcCNg8dMpXrZX0lir7Rj5nRrdU+nYJXE1MNQ8XEv29NeFqFaIO"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:20:44.380Z",
          "created_at": "2023-11-22T13:20:44.380Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.9.2_001",
        "_id": "fleet-uninstall-tokens:f0f540d1-8939-11ee-a3d6-65b5ebccf826",
        "_score": 5.049856,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "f845fe40-8936-11ee-aa3d-eb15ae9ef4d3",
            "token": "RrClbJ0IwCI8TuhSz4p0VahNgiSgJl+RSw7u0ZpLcDwmK/hgSAvLWHvDCGhMPXTdqyCgD+TXdPKRPFpD/Gs+VpSRN5SGOh8e58Bk2AbN1IG6+bgynRHISb2bZgYigFrnsugP3GgK6GGu9Nf+IKGgfNVpqM3DZLIzcQAiAooK"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:20:44.380Z",
          "created_at": "2023-11-22T13:20:44.380Z"
        }
      }
    ]
  }
}

kpollich · 2023-11-22T13:26:49Z

I'll try an upgrade from 8.11.0 -> 8.11.1 to see if anything changes there, as the uninstall tokens will already exist during the stack upgrade process.

kpollich · 2023-11-22T13:54:34Z

Upgrade from 8.11.0 -> 8.11.1 shows the same behavior. Decryption works as expected, encryption key persists across stack upgrades

8.11.0 cluster

[2023-11-22T13:37:35.147+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: AfazsxBuG/SEafuxGNFojYIEpCDGT62rwOLh2qaHYBE=

{
  "took": 10,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 4.8225827,
    "hits": [
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:6793e4a0-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "66f95250-893d-11ee-9258-11a7a4131f91",
            "token": "/Ob5GprdLaCu1pdW1HQyAQXruTmanG1dAfHOEyoXBiGrpGPA+kJd4Z4I8iKjJffsT93TVrvV+2eU4CjwQIEkr3AJAMs55KO/wQNP0T95qzU/OchZSobQulw81pZY7asYd3H3CzCGVFqysfEFZnxhNGthgoLQR2KNMZQy9OmT"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:45:31.882Z",
          "created_at": "2023-11-22T13:45:31.882Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:80a74400-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "804044d0-893d-11ee-9258-11a7a4131f91",
            "token": "sG2QOoZMJSWDp5O18kN/76OMGHk+9C6gxTtYfynS/M/ktMC3w+42mKCDFnEtl8coTxCDE++MroGxZ/lUOal5h3mK0UllFcGZpzkotnLfiKXEaN/808XTtkBwG2iHzhhMiYzPitRxnwmXc1iQvVOl6hldoFAOSwVOoK1V8Osc"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:46:13.952Z",
          "created_at": "2023-11-22T13:46:13.952Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:8c994790-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "8c45f770-893d-11ee-9258-11a7a4131f91",
            "token": "yoNIHVRpw6gE+1nKNa9VcHi9RvVMrecCjXx6kh+onKwcz7m6i3KqYrnGsU5bUVJ43dJPY79CDvQGadlC3NDZXDVagEIOXXgPwgdCns73Psd4sORYBMxLMC3IwoDfDOKMclBrX1TwL0Utjdn7WWXxacUDX4Bq1XEiSqiLRmWw"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:46:33.993Z",
          "created_at": "2023-11-22T13:46:33.993Z"
        }
      }
    ]
  }
}

8.11.1

[2023-11-22T13:48:48.231+00:00][INFO ][plugins.encryptedSavedObjects] Hashed 'xpack.encryptedSavedObjects.encryptionKey' for this instance: AfazsxBuG/SEafuxGNFojYIEpCDGT62rwOLh2qaHYBE=

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 3,
      "relation": "eq"
    },
    "max_score": 4.8225827,
    "hits": [
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:6793e4a0-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "66f95250-893d-11ee-9258-11a7a4131f91",
            "token": "/Ob5GprdLaCu1pdW1HQyAQXruTmanG1dAfHOEyoXBiGrpGPA+kJd4Z4I8iKjJffsT93TVrvV+2eU4CjwQIEkr3AJAMs55KO/wQNP0T95qzU/OchZSobQulw81pZY7asYd3H3CzCGVFqysfEFZnxhNGthgoLQR2KNMZQy9OmT"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:45:31.882Z",
          "created_at": "2023-11-22T13:45:31.882Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:80a74400-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "804044d0-893d-11ee-9258-11a7a4131f91",
            "token": "sG2QOoZMJSWDp5O18kN/76OMGHk+9C6gxTtYfynS/M/ktMC3w+42mKCDFnEtl8coTxCDE++MroGxZ/lUOal5h3mK0UllFcGZpzkotnLfiKXEaN/808XTtkBwG2iHzhhMiYzPitRxnwmXc1iQvVOl6hldoFAOSwVOoK1V8Osc"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:46:13.952Z",
          "created_at": "2023-11-22T13:46:13.952Z"
        }
      },
      {
        "_index": ".kibana_ingest_8.11.0_001",
        "_id": "fleet-uninstall-tokens:8c994790-893d-11ee-9258-11a7a4131f91",
        "_score": 4.8225827,
        "_source": {
          "fleet-uninstall-tokens": {
            "policy_id": "8c45f770-893d-11ee-9258-11a7a4131f91",
            "token": "yoNIHVRpw6gE+1nKNa9VcHi9RvVMrecCjXx6kh+onKwcz7m6i3KqYrnGsU5bUVJ43dJPY79CDvQGadlC3NDZXDVagEIOXXgPwgdCns73Psd4sORYBMxLMC3IwoDfDOKMclBrX1TwL0Utjdn7WWXxacUDX4Bq1XEiSqiLRmWw"
          },
          "type": "fleet-uninstall-tokens",
          "references": [],
          "managed": false,
          "coreMigrationVersion": "8.8.0",
          "updated_at": "2023-11-22T13:46:33.993Z",
          "created_at": "2023-11-22T13:46:33.993Z"
        }
      }
    ]
  }
}

kpollich · 2023-11-27T17:46:52Z

We've validated a workaround that will unblock Fleet:

To unblock Fleet, you'll need to blow away and regenerate all uninstall tokens for your agent policies. It's recommended to keep a record of the existing uninstall tokens somewhere safe, in case they are present elsewhere in your environment or eventually appear in logs when an agent is uninstalled.

Perform the following actions, all requests are expected to be made in Kibana dev tools:

# Create a role with CRUD access to system indices
POST _security/role/system-index-superuser
{
  "indices": [
    {
      "names": [
        "*"
      ],
      "privileges": [
        "all"
      ],
      "allow_restricted_indices": true
    }
  ]
}

# Create a user and assign it that role
POST _security/user/temp_user
{
  "password": "temp_password",
  "roles": [
    "superuser",
    "system-index-superuser"
  ]
}

Then, in another browser session (incognito/private window is easiest) log into Kibana as that user and open dev tools while logged in as the temporary user with elevated permissions.

# Check the list of Fleet uninstall tokens, note the results somewhere like a .txt file
GET .kibana_ingest/_search?q=type:fleet-uninstall-tokens

# Delete all uninstall tokens
POST .kibana_ingest/_delete_by_query
{
  "query": {
    "match": { 
      "type": "fleet-uninstall-tokens"
    }
  }
}

# Rerun Fleet setup to generate new tokens
POST kbn:/api/fleet/setup

# Fetch the new tokens and compare them against the previous results to make sure new
# values have been generated
GET .kibana_ingest/_search?q=type:fleet-uninstall-tokens

Afterwards, the temp_user can be deleted from the initial Kibana browser session.

DELETE _security/user/temp_user

We're also working on an fix for 8.11.2 that will prevent Fleet from being blocked by encryption errors and improve the error logging for the uninstall token service to see if we can get a better idea of what's happening here: #171998.

The root cause is still unclear, and we're continuing to investigate.

kpollich · 2023-12-12T16:13:29Z

Hi all. The security team is tracking this internally, and the workaround above should unblock anyone that runs into the issue as described. I'm closing this in favor of the internal issue in the security team's repo.

huangharry · 2024-01-11T01:55:35Z

Hi Kpollich,
Does it fixed on ELK 8.11.2 or 8.11.3?

kpollich · 2024-01-11T13:26:49Z

Hi @huangharry - if you're experiencing issues with Fleet specifically the workaround in this comment should unblock you.

Are you experiencing issues related to encrypted saved objects in other plugins? Happy to take a look at any error logs, screenshots, or other information you could provide 🙏

kpollich added Team:Fleet Team label for Observability Data Collection Fleet team Team:Defend Workflows “EDR Workflows” sub-team of Security Solution labels Nov 21, 2023

kpollich self-assigned this Nov 21, 2023

kpollich closed this as not planned Won't fix, can't repro, duplicate, stale Dec 12, 2023

cxlashey mentioned this issue Feb 8, 2024

Saving Agent Policy fails with "Cannot read existing Message Signing Key pair" #176528

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 #171630

[Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 #171630

kpollich commented Nov 21, 2023

elasticmachine commented Nov 21, 2023

elasticmachine commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 22, 2023

kpollich commented Nov 22, 2023

kpollich commented Nov 22, 2023

kpollich commented Nov 27, 2023

kpollich commented Dec 12, 2023

huangharry commented Jan 11, 2024

kpollich commented Jan 11, 2024

[Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 #171630

[Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 #171630

Comments

kpollich commented Nov 21, 2023

Examples

During Fleet Setup

In the Rules and Alerts Plugin

Investigation

Open questions

elasticmachine commented Nov 21, 2023

elasticmachine commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 21, 2023

kpollich commented Nov 22, 2023

8.9.2 cluster

8.11.1 cluster

kpollich commented Nov 22, 2023

kpollich commented Nov 22, 2023

8.11.0 cluster

8.11.1

kpollich commented Nov 27, 2023

kpollich commented Dec 12, 2023

huangharry commented Jan 11, 2024

kpollich commented Jan 11, 2024