Skip to content

Rephrase agent advisory and gate by CLI version#919

Merged
codyde merged 1 commit into
masterfrom
agent-advisory-version-gate
May 26, 2026
Merged

Rephrase agent advisory and gate by CLI version#919
codyde merged 1 commit into
masterfrom
agent-advisory-version-gate

Conversation

@codyde
Copy link
Copy Markdown
Collaborator

@codyde codyde commented May 26, 2026

Summary

The previous agent advisory was flagged as prompt-injection by Claude and other agents that scan for directive framing. Users reported the warning firing on roughly every other Railway command during agent sessions.

Two changes:

1. Rephrase to neutral status

Before:

IMPORTANT: Railway agent tooling not detected.

Ask the user if they would like this agent to run `railway setup agent -y`
to configure the Railway agent tooling (agent skills and MCP configurations).

Your agent will work much more effectively in Railway with these installed.

After:

Railway agent tooling (skills + MCP) isn't installed.
Run `railway setup agent` to configure it.
  • No "ask the user," no "this agent" framing — that combination is the textbook prompt-injection pattern.
  • Drops -y from the suggested command so the user stays in the loop on the install.

2. Gate by CLI version instead of 24h timer

The advisory now fires once per installed CLI version, instead of once per 24h per command. Upgrading the CLI re-arms it exactly once. This eliminates the "fired again because you used a different subcommand" surprise.

State file changes

~/.railway/agent-state.json gains advisory.lastShownCliVersion. The old per-command advisories map is dropped from the schema:

{
  "setup": { ... },
  "advisory": {
    "lastShownCliVersion": "4.62.0",
    "lastShownAt": "2026-05-26T..."
  }
}

Existing state files deserialize cleanly (extra fields ignored). Users on the old schema will see the new advisory at most one more time after upgrading.

Behavior matrix

State Pre Post
First run, agent env, setup not done shown shown
Same CLI version, next command suppressed for 24h, then shown suppressed
25h later, same CLI version shown again suppressed
After CLI upgrade shown again shown once
After railway setup agent suppressed permanently suppressed permanently
RAILWAY_AGENT_ADVISORY=0 suppressed suppressed

Test plan

  • Fresh agent env (e.g. CLAUDECODE=1), no ~/.railway/agent-state.json: advisory shows, state file written with lastShownCliVersion matching cargo pkgid.
  • Re-run same command at same CLI version: advisory suppressed.
  • Bump version, re-run: advisory shows again, state updated.
  • RAILWAY_AGENT_ADVISORY=0: suppressed regardless of state.
  • railway setup agent then any other command: suppressed regardless of state.
  • Old state file with advisories: { "up": { ... } } still loads and the new advisory shows at most once.

🤖 Generated with Claude Code

The previous advisory text was flagged as prompt-injection by Claude
and other agents that scan for directive framing. Reports from users
showed the warning firing on every other Railway command in agent
sessions.

Two changes here:

1. Rephrase the message to be neutral status, not a directive aimed
   at the agent. Drops "Ask the user if they would like this agent
   to run ..." and the `-y` non-interactive flag from the suggested
   command. The new message states a fact and offers a remediation;
   the agent decides what to do with it.

2. Replace the 24-hour per-command throttle with a per-CLI-version
   gate. The advisory now fires once per installed CLI version
   instead of every 24 hours per command. Upgrading the CLI re-arms
   it exactly once.

State file (`~/.railway/agent-state.json`) gains
`advisory.lastShownCliVersion`. The old `advisories` map is dropped;
existing state files deserialize cleanly (extra fields ignored) and
will see the advisory at most one more time after upgrade.

`RAILWAY_AGENT_ADVISORY=0` still disables the advisory entirely, and
`railway setup agent` still suppresses it permanently via
`record_setup_complete`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@codyde codyde added the release/minor Author minor release label May 26, 2026
@codyde codyde merged commit 62d7350 into master May 26, 2026
6 of 7 checks passed
@codyde codyde deleted the agent-advisory-version-gate branch May 26, 2026 16:52
codyde added a commit that referenced this pull request Jun 4, 2026
…-browser-login

* origin/master: (28 commits)
  chore: Release railwayapp version 5.0.0
  feat(sandbox): fork, templates, and `--variable`/`--env-file` on create (#933)
  chore: Release railwayapp version 4.68.0
  feat(volume): include modified time in files JSON (#931)
  chore: Release railwayapp version 4.67.0
  Add service source connection support (#934)
  chore: Release railwayapp version 4.66.2
  Make GraphQL HTTP timeout configurable via RAILWAY_HTTP_TIMEOUT (#932)
  chore: Release railwayapp version 4.66.1
  feat(volume): show status and scheduled deletion date in volume list (#928)
  SSH Command: Handle Identity Files (#926)
  chore: Release railwayapp version 4.66.0
  feat(sandbox): `railway sandbox` commands (create/list/ssh/exec/destroy) (#925)
  chore: Release railwayapp version 4.65.0
  SSH Agent Support, `russh` edition. (#915)
  chore: Release railwayapp version 4.64.0
  chore: Release railwayapp version 4.63.0
  Rephrase agent advisory and gate by CLI version (#919)
  Forward --remote to setup agent in cli.new installer (#918)
  chore: Release railwayapp version 4.62.0
  ...

# Conflicts:
#	src/consts.rs
#	src/util/mod.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release/minor Author minor release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant