gcp: fix test configs

[stefans-elastic]

Proposed commit message

gcp: fix credentials_json quoting in Handlebars templates

GCP package templates wrapped credentials_json in literal single quotes
('{{credentials_json}}'). When the credentials JSON value contains
embedded newlines — as PEM-encoded private keys do — the rendered
Agent-policy YAML places those newlines inside a single-quoted flow
scalar, which the eemeli/yaml parser Fleet now uses rejects:

Missing closing 'quote at line N, column N

Replace the manual single-quote wrapping with the Fleet-provided
{{escape_string credentials_json}} helper across all 18 affected
templates (gcp-pubsub.yml.hbs and stream.yml.hbs variants). The helper
inspects the value at render time: safe scalars are left unquoted;
values containing newlines or special characters are emitted as
double-quoted strings with \n escapes.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

• Closes [Stack 9.5.0-SNAPSHOT] [gcp] Failing test daily: system test: pubsub in gcp.loadbalancing_logs #18725

Screenshots

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

This is not the correct fix; there is a template error that the new Fleet YAML handling code has uncovered. The correct fix is, for example:

diff --git a/packages/gcp/data_stream/loadbalancing_logs/agent/stream/gcp-pubsub.yml.hbs b/packages/gcp/data_stream/loadbalancing_logs/agent/stream/gcp-pubsub.yml.hbs
index d69cec05b2..e3f22e4aa1 100644
--- a/packages/gcp/data_stream/loadbalancing_logs/agent/stream/gcp-pubsub.yml.hbs
+++ b/packages/gcp/data_stream/loadbalancing_logs/agent/stream/gcp-pubsub.yml.hbs
@@ -5,7 +5,7 @@ subscription.name: {{subscription_name}}
 credentials_file: {{credentials_file}}
 {{/if}}
 {{#if credentials_json}}
-credentials_json: '{{credentials_json}}'
+credentials_json: {{escape_string credentials_json}}
 {{/if}}
 {{#if alternative_host}}
 alternative_host: {{alternative_host}}

This needs to be applied to all the .hbs files in the integration. Suggest the following commit message:

gcp: use escape_string for credentials_json in templates

The Handlebars templates wrap credentials_json in literal single
quotes ('{{credentials_json}}'). When the value contains newlines,
the rendered YAML has literal newlines inside a single-quoted flow
scalar, which eemeli/yaml rejects during parse:

    Missing closing 'quote at line 14, column 51

Fleet provides an escape_string helper that detects newlines and
switches to double-quoted form with \n escapes. Replace the manual
single-quote wrapping with {{escape_string credentials_json}} in
all 24 affected templates across gcp, gcp_pubsub, gcp_metrics,
gcp_vertexai, and kubernetes/audit_logs.

Fixes #18722
Fixes #18723
Fixes #18724
Fixes #18725

[github-actions]

TL;DR

This failed in Buildkite’s pipeline-upload bootstrap step, not in package/test code from this PR. The buildkite-agent pipeline upload .buildkite/pipeline.yml command ran without an active Buildkite agent context (Missing agent), so the immediate action is to re-run the build and verify agent availability for this pipeline.

Remediation

• Re-run Buildkite build 42467; this looks like a transient agent/bootstrap infrastructure failure.
• If it reproduces, check the Buildkite agent fleet/queue for the integrations pipeline and confirm the upload step is executed by a bootstrapped agent (job API token/env present), then re-run.

Investigation details

Root Cause

The only failing job is :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml, and its log exits immediately with Missing agent. That indicates the failure occurred in Buildkite agent bootstrap/execution context before repository validation/tests began.

No failing assertion, compile error, or package-level test output was produced, so this does not currently point to a source-code regression in the PR diff.

Evidence

• Build: https://buildkite.com/elastic/integrations/builds/42467
• Job/step: :pipeline::arrow_up: Upload Pipeline: .buildkite/pipeline.yml
• Key log excerpt:

Missing agent. See: `buildkite-agent bootstrap --help`

• Failure summary context:
  • Command: buildkite-agent pipeline upload .buildkite/pipeline.yml
  • Exit status: 1

Verification

• Not run: local package/unit/integration tests (the CI failure occurred before test execution started).

Follow-up

If the retry fails with the same message, capture the full agent bootstrap context (queue/agent assignment and env for that step) to confirm whether this is an agent provisioning/config issue specific to this pipeline.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #18819 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• gcp: fix test configs #18819 pull_request_read: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[efd6]

@stefans-elastic Please address #18819 (review).

[stefans-elastic]

@stefans-elastic Please address #18819 (review).

sorry, somehow I messed the original comment.
I've reverted my original fix and implemented a new one in accordance to #18819 (review)

[efd6]

Thanks

[efd6]

@stefans-elastic Please make sure you paste in commit message along the lines of what is in #18819 (review) (it doesn't have to be exactly that, but it should be prose and explain the issue and change as that does).

[stefans-elastic]

@stefans-elastic Please make sure you paste in commit message along the lines of what is in #18819 (review) (it doesn't have to be exactly that, but it should be prose and explain the issue and change as that does).

good note (I would have forgotten), thanks
Done!

[stefans-elastic]

pinging @elastic/obs-ds-hosted-services and @elastic/obs-infraobs-integrations for code owner review

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1a86159

History

• 💚 Build #42708 succeeded 68a0997
• 💚 Build #42631 succeeded 2499f30
• 💔 Build #42467 failed d311820
• 💔 Build #42465 failed 7e58d30
• 💚 Build #42323 succeeded e1878b4

[elastic-vault-github-plugin-prod]

Package gcp - 2.50.1 containing this change is available at https://epr.elastic.co/package/gcp/2.50.1/