Stream data sets through a Zowe SDK sidecar by Tannex · Pull Request #6 · Tannex/cq

Tannex · 2026-07-14T19:57:44Z

Summary

Replaces the per-call `zowe` CLI wrapper with a small Node sidecar built on the official Zowe SDK, and layers cq's DSN access on top of it:

Auth reuse, not reimplementation — the sidecar resolves the user's existing Zowe configuration (team config, OS secure credential store, `zowe auth login` tokens) through the SDK's `ProfileInfo` API, the same code path the Zowe CLI and Zowe Explorer use. cq never sees credentials.
Streaming — data sets flow over one warm z/OSMF HTTPS session straight into the record decoder: no per-call Node startup, no temporary download file, and early exits cancel the transfer mid-stream.
Concurrent COPY resolution — `dsnSearchPath` libraries and sibling COPY members are probed in parallel over the multiplexed sidecar protocol; search order still decides which copy wins.
Server-side bounds for `-max` — with `-max N` (and no `-where`), the sidecar requests just N records via `X-IBM-Record-Range` in record mode, validating every record's length prefix against the layout and transparently restarting as a plain binary stream on any mismatch, so the bound can only save work, never change output.
`--verbose` — config resolution, every z/OSMF request with byte counts and timings, resolver decisions, and record counts on stderr; stdout stays clean JSON.

The zowe CLI wrapping path is fully removed. The sidecar starts lazily on first DSN access (`cq-zowe-sidecar` from PATH by default, `"sidecar"` in config.json to override), so purely local runs don't need it.

Testing

Full Go suite under `-race`, including a fake-sidecar helper process covering the protocol (mux interleaving, cancel, error paths, record-hint pass-through).
Real sidecar + real `@zowe/imperative` driven end-to-end against a fake z/OSMF (TLS, basic auth from team config and OS keyring, record mode, record ranges): ranged reads, range-rejected fallback, and record-length-mismatch fallback all verified on the wire.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

Probe every DSNSearchPath library in parallel per member, keeping the result from the earliest library in search order, and prefetch the distinct COPY members of each expansion level concurrently so slow per-member Zowe requests overlap. Concurrent zowe invocations are bounded to eight, and resolved members and failed lookups are both cached for the command. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

Log the loaded config and search path, each zowe invocation with byte counts and durations, resolver decisions (which library won, cache hits, misses), the record shape, and the decoded record count. The debug logger writes to stderr only when --verbose is set, so JSON on stdout stays clean for piping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

Replace the one-zowe-CLI-process-per-data-set transport with an optional long-lived Node sidecar that resolves the user's existing Zowe configuration (team config, secure credential store, tokens) through the official Zowe SDK's ProfileInfo API and serves data set reads over the z/OSMF REST API. cq and the sidecar speak newline-delimited JSON over stdio, with multiplexed requests, base64-chunked streaming, and cancel for early exits, so records decode straight off the wire with no temporary file and copybook probes share one warm process. Select it with --sidecar 'node sidecar/zowe-sidecar.js' or the Sidecar config field; the zowe CLI remains the default transport behind a new zoweTransport seam. The sidecar owns credentials end to end; cq never sees them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

Standalone @zowe/imperative has no credential manager, so any Zowe team config with secure fields (the zowe CLI default) failed with 'Failed to initialize secure credential manager'. Depend on @zowe/secrets-for-zowe-sdk and pass its keyring to ProfileInfo through ProfileCredentials.defaultCredMgrWithKeytar — the same keyring the zowe CLI itself uses (macOS Keychain, Windows Credential Manager, libsecret). Plain-text configs keep working without the native module, and a failed store load now reports what to do about it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

Remove the per-call zowe CLI wrapper: all DSN access now goes through the Zowe SDK sidecar, started lazily on first data set access so purely local runs never pay for it. The default command is cq-zowe-sidecar from PATH (npm install -g ./sidecar); the config file's sidecar field overrides it. With -max N and no -where filter, the download request carries the record count and length, and the sidecar asks z/OSMF for just those records via X-IBM-Record-Range in record mode. Each record's length prefix is validated against the layout's record length; any mismatch, partial trailer, or HTTP error on the ranged request transparently restarts the transfer as a plain binary stream (skipping bytes already delivered), so the bound can only save work, never change output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

The IBM z/OSMF reference confirms SSS,NNN offsets are 0-based, record mode prefixes are 4-byte big endian lengths excluding the prefix, and a range matching no records returns an exception rather than an empty response — verified end-to-end that the fallback turns that exception into an empty result instead of an error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

process.stdout.write() never blocks: when the pipe to cq is full it queues the write on the heap and returns false, and the z/OSMF socket kept delivering, so a consumer slower than the mainframe (cq piped into a throttled jq, a pager, a slow disk) ballooned the sidecar's memory with the entire un-consumed remainder of the data set. Pause the HTTP response whenever a frame write reports a full pipe and resume it on stdout drain, in both the plain and record-ranged streaming paths. Measured with a 160MB data set against a rate-limited consumer: peak sidecar RSS drops from 666MB (scaling with data set size) to a flat 143MB one-burst plateau that holds at 4x the data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UjWnBCSp1WmbKwiWYGwKqK

- Fail reads on a closed sidecar stream with io.ErrClosedPipe instead of blocking forever once the buffered bytes are consumed - Resolve COPY members as soon as the winning library is known and cancel later in-flight probes (fetchCopybook now takes a context), so a slow irrelevant library never delays or blocks an earlier match - Parse quoted executables and arguments in the sidecar command through splitCommandSpec, shared with the config editor spec - Give ZOWE_OPT_* environment variables the zowe CLI's precedence over profile properties, and honor ZOWE_OPT_ZOSMF_PROFILE profile selection - Reject Basic-auth profiles that have a user but no password up front instead of sending "user:undefined" - Bound non-200 error body buffering to a 16 KiB prefix with a timeout - Fall back to a plain transfer when a ranged request or response fails mid-stream, resuming after the records already delivered - Require Node >=20.9.0, matching the locked @zowe/imperative engine Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014CmkgLDN4iT6z2B23YuVQf

claude and others added 9 commits July 14, 2026 14:35

refactor config keys to camelCase

36aa70e

refactor headers

38a1c8a