v0.6.8 - #1352
Conversation
* Claude update * Fix file format * Recovering missing fields * LocalDate as date
…rs (#1345) * Document GCP APIs and DWD scope strings for Google Workspace connectors. Customers need explicit lists of APIs to enable and comma-separated OAuth scope URLs for Domain-wide Delegation grants, including a superset for shared service accounts. Co-authored-by: Cursor <cursoragent@cursor.com> * Merge DWD scope CSV into Required OAuth Scopes on connector pages. Avoid a separate Domain-wide Delegation section; keep the paste-ready scope string under the existing OAuth scopes heading. Co-authored-by: Cursor <cursoragent@cursor.com> * Revert docs/README.md Google Workspace table to short scope list. Keep detailed API and DWD scope documentation on the Google Workspace connector pages only. Co-authored-by: Cursor <cursoragent@cursor.com> * Classify Google setup errors and slim troubleshooting docs. Return specific X-Psoxy-Error codes and sanitized bodies for API-not-enabled and OAuth setup failures; document error-to-cause mapping as a skimmable list with dummy log examples. Co-authored-by: Cursor <cursoragent@cursor.com> * Rename SOURCE_DWD_NOT_GRANTED and inject GoogleApiSetupErrorInterpreter. Use SOURCE_AUTHORIZATION_NOT_GRANTED for cross-source admin-consent failures; wire ObjectMapper via Dagger constructor injection. Co-authored-by: Cursor <cursoragent@cursor.com> * Document raw Google error signals for older proxy versions. Show parsed log/response fragments that match GoogleApiSetupErrorInterpreter, with legacy X-Psoxy-Error fallbacks. Co-authored-by: Cursor <cursoragent@cursor.com> * Note that OAuth scope mismatch can surface as 401 Unauthorized. Document indistinguishable 401 for missing DWD vs wrong scopes; broaden SOURCE_AUTHORIZATION_NOT_GRANTED message accordingly. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix OAuth scope hint to reference OAUTH_SCOPES env var. Deployed proxies read scopes from the environment via EnvVarsConfigService, not config.yaml. Co-authored-by: Cursor <cursoragent@cursor.com> * Log OAuth token exchange failures at ERROR severity. 401 and other oauth2.googleapis.com/token failures now use SEVERE so they appear as errors in Cloud Logging. Co-authored-by: Cursor <cursoragent@cursor.com> * Always log connection-setup IOExceptions at SEVERE. Drop Google-specific oauth2.googleapis.com/token check from ApiDataRequestHandler; scope hint only when setup error interpreter matches. Co-authored-by: Cursor <cursoragent@cursor.com> * Remove GoogleApiSetupErrorInterpreter; keep docs-only troubleshooting. Roll back Google-specific error classification code and rewrite troubleshooting around log signals mapped to setup conditions. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* Updated test * Updated rules
* Fix String.format placeholders in config cache retry log.
MessageFormat-style {0} placeholders were passed to String.format, so retry attempt details were never interpolated.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Bump minor Java and Node dependency versions.
Routine maintenance: update Maven property versions and AWS/GCP-adjacent libraries, and confirm npm audit fix reports no vulnerabilities in tool packages.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Upgrade Jackson to 2.22.0 via jackson-bom.
Jackson 2.22 uses patch-less versioning for jackson-annotations; importing the BOM aligns module versions consistently with the main Worklytics codebase.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update Jackson BOM to latest
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jose Lorenzo <jose@worklytics.co>
…eployments (#1347) * Add beta GCP external ALB composition support via api_connector_external_lb_host. Wire ingress and public endpoint URLs from a customer-owned ALB host signal, document the pattern, and teach psoxy-test to call ALB URLs with self-signed TLS. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review: fix ALB IP CLI flags and cacert Promise errors. Extract LB host without regex capture groups, reject invalid --cacert paths asynchronously, and remove Terraform interpolation from the external_lb_base_url description so validate succeeds. Co-authored-by: Cursor <cursoragent@cursor.com> * Address PR review: HTTPS enforcement and simpler IP/path handling. - Detect ALB IP hosts with cidrhost instead of regex - Require https:// for proxy endpoints (Terraform check + psoxy-test) - Simplify request path to pathname + search; warn on --allow-insecure-tls Co-authored-by: Cursor <cursoragent@cursor.com> * Fix CI: assert HTTPS on known external_lb_base_url only. Checking proxy_endpoint_url fails terraform test because the Cloud Function URI is unknown until apply. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix HTTPS check for Terraform <1.12 null short-circuit. Older Terraform evaluates both sides of ||, so startswith(null) failed module tests. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com>
* bump package-lock * improve docs * improve those docs * refactor ALB stuff * comment out external api alb in example * fix IPv6-CIDR parsing in IP allowlists Apache Commons Net SubnetUtils is IPv4-only, so valid IPv6-CIDR entries like /128 were rejected at startup. Co-authored-by: Cursor <cursoragent@cursor.com> * TODO to hide function url in prod; not possible via tf atm * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address PR review comments on ALB example wiring Remove the misleading top-level api_connector_external_lb_host variable; document the commented main.tf binding instead. Clarify troubleshooting placeholders to use cloud-function-name and environment_id_prefix. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
#1351) Co-authored-by: Cursor <cursoragent@cursor.com>
|
NOTE: full PR updating various demo envs to rc-v0.6.8: https://github.com/Worklytics/psoxy-demos/pull/166, with examples of each passing. |
There was a problem hiding this comment.
Pull request overview
This PR merges the v0.6.8 release back to main, adding a new Microsoft OneDrive connector (rules + examples + Java tests), improving GCP external ingress support via an optional external ALB + Cloud Armor composition pattern, and bumping versions/dependencies across the Java and Node tooling.
Changes:
- Add
msft-onedriveconnector specs, rules, docs, and example API responses (including “no app ids” variants) plus Java rules tests. - Improve GCP deployment ergonomics for external ALB frontends: new
ingress_settings/ external endpoint wiring, andpsoxy-testsupport for--allow-insecure-tls/--cacert. - Version bump to
v0.6.8and dependency updates (Maven + npm lockfile), plus small infra/docs cleanups.
Reviewed changes
Copilot reviewed 94 out of 98 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/release/example-copy.sh | Force LF line endings and publish additional helper scripts when copying example repos. |
| tools/psoxy-test/test/utils.test.js | Add tests for HTTPS request option building and TLS flags (--allow-insecure-tls, --cacert). |
| tools/psoxy-test/test/gcp.test.js | Update GCP logs URL behavior and adjust call signature expectations. |
| tools/psoxy-test/test/aws.test.js | Adjust AWS call test expectations for updated request signature. |
| tools/psoxy-test/package-lock.json | Bump/clean transitive npm deps in the test tool. |
| tools/psoxy-test/lib/utils.js | Add TLS-aware HTTPS request option builder and propagate TLS options into requests. |
| tools/psoxy-test/lib/gcp.js | Plumb TLS options through GCP test calls and warn when insecure TLS is enabled. |
| tools/psoxy-test/lib/aws.js | Plumb TLS options through AWS test calls and warn when insecure TLS is enabled. |
| tools/psoxy-test/cli-call.js | Add CLI flags for insecure TLS and custom CA trust. |
| tools/init-tfvars.sh | Bump script version to v0.6.8. |
| java/pom.xml | Bump project revision and multiple dependency versions; import jackson-bom. |
| java/impl/cmd-line/pom.xml | Rely on BOM-managed Jackson versions (remove explicit version pins). |
| java/impl/aws/pom.xml | Update AWS Lambda/New Relic dependency versions. |
| java/gateway-core/pom.xml | Update selected dependencies (e.g., commons-codec, httpclient5, json-schema-validator) and rely on Jackson BOM. |
| java/core/src/test/java/co/worklytics/psoxy/rules/zoom/ZoomRulesTests.java | Add a Zoom meeting URL example with query params. |
| java/core/src/test/java/co/worklytics/psoxy/rules/msft/OneDriveTests.java | Add Java rules test coverage for OneDrive default rules. |
| java/core/src/test/java/co/worklytics/psoxy/rules/msft/OneDrive_NoAppIds_Tests.java | Add Java rules test coverage for OneDrive “no app ids” rules. |
| java/core/src/test/java/co/worklytics/psoxy/gateway/NetworkSecurityUtilsTest.java | Add IPv6 allowlist test coverage. |
| java/core/src/test/java/co/worklytics/psoxy/gateway/impl/ApiDataRequestHandlerTest.java | Update expected proxy version string to v0.6.8. |
| java/core/src/main/java/co/worklytics/psoxy/rules/msft/PrebuiltSanitizerRules.java | Register OneDrive rules in MSFT prebuilt rules map. |
| java/core/src/main/java/co/worklytics/psoxy/gateway/ProxyConstants.java | Bump Java proxy version constant to v0.6.8. |
| java/core/src/main/java/co/worklytics/psoxy/gateway/NetworkSecurityUtils.java | Add IPv6 CIDR matching support for IP allowlisting. |
| java/core/src/main/java/co/worklytics/psoxy/gateway/impl/CachingConfigServiceDecorator.java | Improve transient-retry logging and cache refresh behavior logging. |
| java/core/src/main/java/co/worklytics/psoxy/gateway/impl/ApiDataRequestHandler.java | Update guidance message for OAuth scope configuration mismatch. |
| java/core/pom.xml | Rely on BOM-managed Jackson versions; bump nimbus-jose-jwt. |
| infra/modules/worklytics-connectors-msft-365/variables.tf | Document supported msft_365_connector_settings keys including OneDrive example IDs. |
| infra/modules/worklytics-connectors-google-workspace/main.tf | Fix local alignment/formatting in locals block. |
| infra/modules/worklytics-connector-specs/variables.tf | Document supported msft_365_connector_settings keys including OneDrive example IDs. |
| infra/modules/worklytics-connector-specs/msft-365.tf | Add msft-onedrive connector spec (beta) including example calls and roles. |
| infra/modules/worklytics-connector-specs/main.tf | Add locals to support OneDrive example IDs used in generated example API calls. |
| infra/modules/google-workspace-dwd-connection/main.tf | Fix local alignment/formatting in locals block. |
| infra/modules/gcp-proxy-bulk/variables.tf | Minor formatting change in variable type declaration. |
| infra/modules/gcp-proxy-api/variables.tf | Add ingress_settings and deprecated external_lb_base_url for ALB-based example URLs. |
| infra/modules/gcp-proxy-api/main.tf | Support external ALB endpoint URLs for tests/TODOs; keep Pub/Sub push to Cloud Function URI; configurable ingress. |
| infra/modules/gcp-host/variables.tf | Add api_connector_external_lb_host variable (beta) for external ALB frontends. |
| infra/modules/gcp-host/main.tf | Wire external ALB host into API connector ingress settings and published endpoint URLs; add safety check requiring allowlist. |
| infra/modules/entra-grant-all-users/main.tf | Update Entra admin URL path for API permissions grant step. |
| infra/examples-dev/gcp/vpc.tf | Add optional (commented) VPC egress composition example. |
| infra/examples-dev/gcp/variables.tf | Tighten environment_name validation and improve allowlist docs for Cloud Armor composition. |
| infra/examples-dev/gcp/msft-365.tf | Update example module source refs to v0.6.8. |
| infra/examples-dev/gcp/msft-365-variables.tf | Document supported msft_365_connector_settings keys including OneDrive example IDs. |
| infra/examples-dev/gcp/main.tf | Add optional external ALB host wiring comments; add tls provider requirement for the ALB PoC composition; update module refs. |
| infra/examples-dev/gcp/google-workspace.tf | Update example module source refs to v0.6.8. |
| infra/examples-dev/gcp/external-api-alb.tf | Add commented composition template for global external ALB + Cloud Armor + TLS (beta). |
| infra/examples-dev/aws/msft-365.tf | Update example module source refs to v0.6.8. |
| infra/examples-dev/aws/msft-365-variables.tf | Document supported msft_365_connector_settings keys including OneDrive example IDs. |
| infra/examples-dev/aws/main.tf | Update example module source refs to v0.6.8. |
| infra/examples-dev/aws/google-workspace.tf | Update example module source refs to v0.6.8. |
| docs/SUMMARY.md | Add navigation entries for GCP external ALB doc and OneDrive connector docs. |
| docs/sources/zoom/zoom.yaml | Allow Zoom meeting query params (occurrence_id, show_previous_occurrences). |
| docs/sources/README.md | Add msft-onedrive to the connector list. |
| docs/sources/microsoft-365/README.md | Add guidance on populating example API call placeholders and include OneDrive in the scopes table. |
| docs/sources/microsoft-365/msft-onedrive/README.md | New OneDrive connector documentation (ID, endpoints, scopes, examples, placeholder population). |
| docs/sources/microsoft-365/msft-onedrive/msft-onedrive.yaml | Add OneDrive rules YAML (doc copy). |
| docs/sources/microsoft-365/msft-onedrive/msft-onedrive_no-app-ids.yaml | Add OneDrive “no app ids” rules YAML (doc copy). |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/users.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/groups.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/list_drives.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/get_drive_delta.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/list_itemActivity.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized/list_driveActivity.json | Add sanitized OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/users.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/groups.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/list_drives.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/get_drive_delta.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/list_itemActivity.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/sanitized_no-app-ids/list_driveActivity.json | Add sanitized (no app ids) OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/users.json | Add original OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/groups.json | Add original OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/list_drives.json | Add original OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/get_drive_delta.json | Add original OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/list_itemActivity.json | Add original OneDrive example response. |
| docs/sources/microsoft-365/msft-onedrive/example-api-responses/original/list_driveActivity.json | Add original OneDrive example response. |
| docs/sources/google-workspace/README.md | Expand Workspace connector setup docs: required APIs, DWD scope strings, and troubleshooting mapping. |
| docs/sources/google-workspace/meet/README.md | Add explicit DWD scope string and required GCP API for Meet connector. |
| docs/sources/google-workspace/google-chat/README.md | Add explicit DWD scope string and required GCP API for Google Chat connector. |
| docs/sources/google-workspace/gmail/README.md | Add explicit DWD scope string and required GCP API for Gmail connector. |
| docs/sources/google-workspace/gemini-in-workspace-apps/README.md | Add explicit DWD scope string and required GCP API for Gemini-in-Workspace connector. |
| docs/sources/google-workspace/gdrive/README.md | Add explicit DWD scope string and required GCP API for Drive connector. |
| docs/sources/google-workspace/directory/README.md | Add explicit DWD scope string and required GCP API for Directory connector. |
| docs/sources/google-workspace/calendar/README.md | Add explicit DWD scope string and required GCP API for Calendar connector. |
| docs/sources/anthropic/claude/example-api-responses/sanitized/chats-response.json | Remove organization_id from sanitized example. |
| docs/sources/anthropic/claude/example-api-responses/sanitized/chat-messages-response.json | Remove organization_id from sanitized example. |
| docs/sources/anthropic/claude/claude.yaml | Remove organization_id from schema. |
| docs/sources/anthropic/claude-enterprise-analytics/example-api-responses/sanitized/users.json | Update sanitized analytics example to new field names/shape. |
| docs/sources/anthropic/claude-enterprise-analytics/example-api-responses/original/users.json | Update original analytics example to new field names/shape. |
| docs/sources/anthropic/claude-enterprise-analytics/claude-enterprise-analytics.yaml | Update schema for newly added metrics fields and renamed tool action counters. |
| docs/guides/psoxy-test-tool.md | Document external ALB testing flags and common errors. |
| docs/gcp/vpc.md | Clarify VPC doc scope (egress) and link to external ALB ingress option. |
| docs/gcp/sbom.json | Update SBOM metadata/version to 0.6.8. |
| docs/gcp/README.md | Add VPC doc link and reference external ALB doc. |
| docs/gcp/authentication-authorization.md | Link to external ALB + Cloud Armor doc as an ingress filtering option. |
| docs/development/README.md | Add external ALB + Cloud Armor doc to development index. |
| docs/development/gcp-private-service-connect.md | Add external ALB option to the connectivity matrix and cross-link docs. |
| docs/development/gcp-external-alb.md | New beta doc describing external ALB + Cloud Armor composition pattern. |
| docs/configuration/ip-allowlisting.md | Expand GCP allowlisting guidance to cover Cloud Armor + external ALB composition and troubleshooting. |
| docs/aws/sbom.json | Update SBOM metadata/version to 0.6.8. |
| CHANGELOG.md | Add unreleased entry documenting the new msft-onedrive connector. |
Files not reviewed (1)
- tools/psoxy-test/package-lock.json: Generated file
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if [ -n "$PATH_TO_MAIN_REPO_ROOT" ] && [[ "${PATH_TO_MAIN_REPO_ROOT: -1}" != "/" ]]; then | ||
| PATH_TO_MAIN_REPO_ROOT="$PATH_TO_MAIN_REPO_ROOT/" | ||
| fi |
| log.log(Level.WARNING, | ||
| "Backend returned empty for config property {0} which was previously set; assuming transient failure and retaining cached value", | ||
| "Backend returned empty for config property %s which was previously set; assuming transient failure and retaining cached value", | ||
| key.name()); |
| log.log(Level.WARNING, | ||
| "Transient failure reloading config property {0}; retaining cached value until next refresh cycle", | ||
| "Transient failure reloading config property %s; retaining cached value until next refresh cycle", | ||
| key.name()); |
| for (int i = fullBytes; i < network.length; i++) { | ||
| network[i] = 0; | ||
| } | ||
| if (remainingBits > 0 && fullBytes < network.length) { | ||
| int mask = 0xFF << (8 - remainingBits); | ||
| network[fullBytes] = (byte) (network[fullBytes] & mask); | ||
| } |
| log.log(Level.WARNING, | ||
| "Backend returned empty for config property {0} which was previously set; assuming transient failure and retaining cached value", | ||
| "Backend returned empty for config property %s which was previously set; assuming transient failure and retaining cached value", | ||
| key.name()); |
| log.log(Level.WARNING, | ||
| "Transient failure reloading config property {0}; retaining cached value until next refresh cycle", | ||
| "Transient failure reloading config property %s; retaining cached value until next refresh cycle", | ||
| key.name()); |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 61 changed files in this pull request and generated 2 comments.
Files not reviewed (1)
- tools/psoxy-test/package-lock.json: Generated file
Comments suppressed due to low confidence (4)
java/core/src/main/java/co/worklytics/psoxy/gateway/impl/CachingConfigServiceDecorator.java:109
- These log messages use
%splaceholders but are passed toLogger#log(Level, String, Object)(MessageFormat-style). As written, the%swill be logged literally andkey.name()will be ignored; wrap the message inString.format(...)so the property name appears in logs.
java/core/src/main/java/co/worklytics/psoxy/gateway/impl/CachingConfigServiceDecorator.java:119 - Same placeholder issue here:
%swon't be substituted when usingLogger#log(Level, String, Object); wrap withString.format(...)(or switch the placeholders back to{0}).
java/core/src/main/java/co/worklytics/psoxy/gateway/NetworkSecurityUtils.java:216 maskToNetworkzeros bytes starting atfullBytesbefore applying the partial-byte mask, which wipes the byte that should retain theremainingBitsof the prefix. This breaks IPv6 CIDRs with prefix lengths not divisible by 8 (e.g. /127, /33, etc.).
tools/release/example-copy.sh:68rm ${EXAMPLE_TEMPLATE_REPO}*.bckis unquoted and will error when no backups exist (unmatched glob). Userm -fand quote the variable while keeping the glob outside the quotes.
| @@ -35,7 +55,7 @@ for file in "${FILES_TO_COPY[@]}" | |||
| do | |||
| if [ -f ${EXAMPLE_TO_COPY_FROM}/${file} ]; then | |||
| assertTrue(utils.isDataAccessIpAllowed("2001:db8:1::1")); | ||
| assertFalse(utils.isDataAccessIpAllowed("2001:db9::1")); | ||
| } | ||
| } |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 60 out of 61 changed files in this pull request and generated 1 comment.
Files not reviewed (1)
- tools/psoxy-test/package-lock.json: Generated file
Comments suppressed due to low confidence (2)
java/core/src/main/java/co/worklytics/psoxy/gateway/NetworkSecurityUtils.java:218
maskToNetwork()zeroes bytes starting atfullBytes, which also clears the partially-covered byte whenprefixLength % 8 != 0. That makes the computed network incorrect for non-/8-aligned IPv6 CIDRs (eg/33), socontains()can return incorrect results.
tools/release/example-copy.sh:16- The usage string now documents
[path-to-main-repo]as optional, butPATH_TO_MAIN_REPO_ROOTis still used unconditionally when copying scripts (egtools/init-example.sh). If the third arg is omitted, the script will attempt to copy from a relativetools/...under the example directory and fail. Default this to the repo root based on the script location when not provided.
| for (SubnetUtils utils : ipv4Cidrs.values()) { | ||
| if (utils.getInfo().isInRange(clientIp)) { | ||
| return true; | ||
| } | ||
| } |
all resolved; dismissing to unblock PR
v0.6.8 back to main
Test Plan
AWS
cd infra/examples-dev/aws ./apply ./test-all.shConfirm everything worked:
GCP
cd infra/examples-dev/gcp ./apply ./test-all.shConfirm everything worked:
Next steps, after that's merged to
main:./tools/release/publish.sh v0.6.8