Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 19 additions & 2 deletions .github/workflows/integration-test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,12 @@ on:
description: 'Azure region'
required: false
type: string
default: 'eastus2'
default: 'centralus'
log_level:
description: 'PowerShell log level (Info, Verbose, Debug)'
required: false
type: string
default: Verbose
skip_teardown:
description: 'Skip teardown (for debugging)'
required: false
Expand All @@ -58,7 +63,12 @@ on:
description: 'Azure region'
required: false
type: string
default: 'eastus2'
default: 'centralus'
log_level:
description: 'PowerShell log level (Info, Verbose, Debug)'
required: false
type: string
default: Verbose
skip_teardown:
description: 'Skip teardown (for debugging)'
required: false
Expand Down Expand Up @@ -114,11 +124,18 @@ jobs:
shell: pwsh
run: |
$skipTeardown = '${{ inputs.skip_teardown }}' -eq 'true'
$logLevel = '${{ inputs.log_level }}'
if ([string]::IsNullOrWhiteSpace($logLevel)) { $logLevel = 'Verbose' }
if ($logLevel -notin @('Info', 'Verbose', 'Debug')) {
throw "Invalid log_level '$logLevel'. Allowed values: Info, Verbose, Debug."
}

$params = @{
SourceResourceGroup = '${{ env.SOURCE_RG }}'
TargetResourceGroup = '${{ env.TARGET_RG }}'
SkuName = '${{ inputs.sku }}'
Location = '${{ inputs.location }}'
LogLevel = $logLevel
PublisherEmail = '${{ secrets.APIM_PUBLISHER_EMAIL }}'
ExtractOutputDir = './extracted-artifacts'
HardDelete = $true
Expand Down
7 changes: 6 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,12 @@ Desktop.ini
# Local testing output (use --output .local-extract for local runs)
.local-extract*/

# Log files for integration tests
# Files for integration tests
tests/integration/all-resource-types/logs/**
tests/integration/all-resource-types/extracted-artifacts*/**
tests/integration/all-resource-types/target-apim.json
tests/integration/all-resource-types/source-apim.json
tests/integration/all-resource-types/source-apim-post-activation.bicep

# Environment variables
.env
Expand All @@ -53,3 +57,4 @@ tests/integration/all-resource-types/logs/**
# Squad: SubSquad activation file (local to this machine)
.squad-workstream
*.d.ts.map
tests/integration/all-resource-types/source-apim-post-activation.json
8 changes: 7 additions & 1 deletion .squad/agents/apimexpert/history.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,5 +98,11 @@ the SDK surface, reference docs, or ad-hoc observation.

**Research output:** `.squad/decisions.md` entry (merged from inbox), full analysis in `specs/sku-upgrade.md`

### 2026-05-13: APIM v1 → v2 SKU Migration Research
### 2026-05-19: `policyRestrictions.scope` grammar

`Microsoft.ApiManagement/service/policyRestrictions@2025-09-01-preview`. Schema says `"Path to the policy document."` but the API validates `scope` as a **relative ARM path to an existing API, operation, or product**.

- Accepted: `/apis/{apiId}`, `/apis/{apiId}/operations/{opId}`, `/products/{productId}`, `""`.
- Classic Developer/Premium SKU only.
- Docs: <https://learn.microsoft.com/rest/api/apimanagement/policy-restriction> · <https://learn.microsoft.com/azure/templates/microsoft.apimanagement/service/policyrestrictions>

6 changes: 6 additions & 0 deletions .squad/agents/securityexpert/history.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,9 @@
Performed a thorough read-only sensitivity audit across all tracked files in preparation for open-source publication. Scanned for secrets/credentials, internal Microsoft URLs, PII, internal comments, internal tool configs, sensitive paths, draft docs, hardcoded Azure resource IDs, and internal dependency references. Findings delivered for compliance sign-off. No live credentials, certificates, or storage keys were found. All Azure GUIDs encountered were either zero-padded placeholders or public Azure built-in role definition IDs. Primary cleanup items: a developer machine path and alias references in `.squad/` history/decisions; one real-looking storage account name in a test fixture.

**Findings Summary:** 2 MEDIUM items, 3 LOW items. Orchestration log: `.squad/orchestration-log/2026-05-19T22-01-securityexpert.md`
- When using PowerShell transcript/trace logging, always pass `-UseMinimalHeader` to `Start-Transcript` to prevent machine/host environment details from being written to logs.
- `Start-Transcript -UseMinimalHeader` keeps machine/host details out of logs.
- **ARM async-operation URLs** (`Azure-AsyncOperation` / `Location`) include `t/c/s/h` query params that act as short-lived bearer credentials. Regex-mask them. <https://learn.microsoft.com/azure/azure-resource-manager/management/async-operations>
- **`x-ms-routing-request-id`** carries `REGION:UTC:GUID` — mask the whole value, not just the GUID. <https://learn.microsoft.com/azure/azure-resource-manager/management/request-limits-and-throttling>
- **PowerShell `Start-Transcript` double-emits native stderr** when paired with `2>&1 | Write-Host`. Either regex-mask in `Protect-LogLine` (so both copies get masked) or redirect the child's stderr to a pipe via `System.Diagnostics.Process` so the transcript never sees the raw line. Both layers together = defense in depth.
- **Do not mask all GUIDs.** Azure built-in role-definition IDs and ARM template hashes are public constants useful for debugging. Anchor secret regex to the path segment, header name, or query-parameter context that makes the value sensitive.
13 changes: 13 additions & 0 deletions .squad/agents/testengineer/history.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,3 +159,16 @@
- History updated with dual-mode package consumption patterns

<!-- Append new learnings here after each session -->

### 2026-05-19: MaskingHelpers — capture child stderr directly

Rewrote [tests/integration/all-resource-types/MaskingHelpers.psm1](tests/integration/all-resource-types/MaskingHelpers.psm1) around `System.Diagnostics.Process` with `RedirectStandardOutput/Error = $true` so the child's raw bytes bypass PowerShell's ErrorRecord promotion and never reach the parent transcript. Per-stream `Start-ThreadJob` readers drain into `ConcurrentQueue[string]`s; main runspace polls every 100 ms and emits through `Protect-LogLine`.

Breaking signature: helpers now take `-Arguments [string[]]` instead of `-Command [scriptblock]`. All four call sites updated.

Gotchas for future PowerShell work:

- `Register-ObjectEvent` Action handlers do not drain reliably while the main runspace is in `Start-Sleep`. Use `Start-ThreadJob` (PS 7+) to bypass the engine event pump.
- `$x = if ($cond) { [List[T]]::new() }` assigns `$null` — PowerShell enumerates the empty list. Use `$x = $null; if ($cond) { $x = ... }`.
- `ProcessStartInfo.StandardOutputEncoding/StandardErrorEncoding` default to OEM on Windows; force UTF-8 or `az --debug` output mangles.

5 changes: 4 additions & 1 deletion src/lib/dependency-graph.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,11 @@ import { DependencyEdge } from '../models/types.js';
export const DEPENDENCY_EDGES: DependencyEdge[] = [
// Tier 1 -> Tier 2 dependencies
{ from: ResourceType.Diagnostic, to: ResourceType.Logger, required: false },
{ from: ResourceType.Diagnostic, to: ResourceType.Workspace, required: false },
{ from: ResourceType.ServicePolicy, to: ResourceType.NamedValue, required: false },
{ from: ResourceType.ServicePolicy, to: ResourceType.PolicyFragment, required: false },
{ from: ResourceType.Api, to: ResourceType.VersionSet, required: false },
{ from: ResourceType.PolicyRestriction, to: ResourceType.Product, required: false },

// Tier 2 -> Tier 3 dependencies
{ from: ResourceType.ProductPolicy, to: ResourceType.Product, required: true },
Expand Down Expand Up @@ -61,6 +63,7 @@ export const DEPENDENCY_EDGES: DependencyEdge[] = [
];

export const TIER_1_RESOURCES: ResourceType[] = [
ResourceType.Workspace,
ResourceType.NamedValue,
ResourceType.Tag,
ResourceType.Gateway,
Expand All @@ -70,7 +73,6 @@ export const TIER_1_RESOURCES: ResourceType[] = [
ResourceType.Group,
ResourceType.PolicyFragment,
ResourceType.GlobalSchema,
ResourceType.PolicyRestriction,
ResourceType.Documentation,
];

Expand All @@ -82,6 +84,7 @@ export const TIER_2_RESOURCES: ResourceType[] = [
];

export const TIER_3_RESOURCES: ResourceType[] = [
ResourceType.PolicyRestriction,
ResourceType.ProductPolicy,
ResourceType.ProductGroup,
ResourceType.ProductTag,
Expand Down
43 changes: 35 additions & 8 deletions src/lib/resource-path.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,20 @@ import { ResourceDescriptor } from '../models/types.js';
import { RESOURCE_TYPE_METADATA, ResourceType } from '../models/resource-types.js';

/**
* Association resource types that represent parent-child relationships
* (not independently publishable resources).
* These are handled specially during publishing via association files
* (apis.json, groups.json) and should not be discovered as individual resources.
* Association resource types that are published by specialized parent publishers.
*
* Product associations are handled by product-publisher using the parent
* product descriptor, so their association files should not be discovered as
* independent resources.
*
* Gateway associations are intentionally excluded from this set because they
* are published via the generic association path in resource-publisher and
* must therefore be discovered from gateways/{gateway}/apis.json.
*/
const ASSOCIATION_TYPES = new Set<ResourceType>([
const PARENT_PUBLISHED_ASSOCIATION_TYPES = new Set<ResourceType>([
ResourceType.ProductApi,
ResourceType.ProductGroup,
ResourceType.ProductTag,
ResourceType.GatewayApi,
]);

const SUPPORTED_SPECIFICATION_EXTENSIONS = new Set([
Expand Down Expand Up @@ -369,6 +373,11 @@ export function parseArtifactPath(
return undefined;
}

const workspaceContainer = parseWorkspaceContainerDescriptor(fileName, workspace);
if (workspaceContainer) {
return workspaceContainer;
}

// Try to match against each resource type's pattern
for (const [typeKey, metadata] of Object.entries(RESOURCE_TYPE_METADATA)) {
const type = typeKey as ResourceType;
Expand All @@ -379,8 +388,8 @@ export function parseArtifactPath(

// Skip association resource types — these are handled specially during publishing
// via their parent's association files (apis.json, groups.json)
if (ASSOCIATION_TYPES.has(type)) {
return undefined;
if (PARENT_PUBLISHED_ASSOCIATION_TYPES.has(type)) {
continue;
}

const nameParts = parseTemplatePath(
Expand All @@ -396,6 +405,24 @@ export function parseArtifactPath(
return undefined;
}

function parseWorkspaceContainerDescriptor(
fileName: string,
workspace?: string
): ResourceDescriptor | undefined {
// Workspace container descriptors are stored at:
// workspaces/{workspace}/workspaceInformation.json
// and are not workspace-scoped children. Return a top-level Workspace
// descriptor so publish can create the container before workspace children.
if (fileName === 'workspaceInformation.json' && workspace) {
return {
type: ResourceType.Workspace,
nameParts: [workspace],
};
}

return undefined;
}

/**
* Parse a changed artifact file path into a ResourceDescriptor.
*
Expand Down
8 changes: 8 additions & 0 deletions src/models/resource-types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ export enum ResourceType {
GraphQLResolverPolicy = 'GraphQLResolverPolicy',
/** MCP (Model Context Protocol) server configuration per API. Singleton per API. */
McpServer = 'McpServer',
/** Premium/PremiumV2 workspace container. */
Workspace = 'Workspace',
}

/**
Expand Down Expand Up @@ -278,4 +280,10 @@ export const RESOURCE_TYPE_METADATA: Record<ResourceType, ResourceTypeMetadata>
infoFile: 'mcpServerInformation.json',
supportsGet: true,
},
[ResourceType.Workspace]: {
armPathSuffix: 'workspaces/{0}',
artifactDirectory: 'workspaces/{0}',
infoFile: 'workspaceInformation.json',
supportsGet: true,
},
};
58 changes: 54 additions & 4 deletions src/services/api-publisher.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,22 @@ export async function publishApi(
}

// Step 2: Find and publish revisions in numeric order
await publishApiRevisions(client, store, context, descriptor, config);
const publishedRevisionCount = await publishApiRevisions(client, store, context, descriptor, config);

// Step 2b: Align root API only when source marks it as current.
// Source of truth is properties.isCurrent in root apiInformation.json.
if (publishedRevisionCount > 0 && rootResult.isCurrent === true) {
const alignResult = await alignActiveRevisionWithSource(
client,
store,
context,
descriptor,
config
);
if (alignResult.status !== 'success') {
return alignResult;
}
}

// Step 3: Publish child resources in parallel
// When a spec was imported, operations and schemas are auto-created by APIM
Expand Down Expand Up @@ -105,6 +120,11 @@ function getImportFormat(specFormat: string, _apiType?: string): string | undefi
interface RootApiResult {
status: 'success' | 'skipped';
specImported: boolean;
isCurrent?: boolean;
}

interface PublishRootApiOptions {
includeSpecification?: boolean;
}

/**
Expand All @@ -118,7 +138,8 @@ async function publishRootApi(
store: IArtifactStore,
context: ApimServiceContext,
descriptor: ResourceDescriptor,
config: PublishConfig
config: PublishConfig,
options?: PublishRootApiOptions
): Promise<RootApiResult & ResourcePublishResult> {
let json = await store.readResource(config.sourceDir, descriptor);
if (!json) {
Expand All @@ -132,10 +153,14 @@ async function publishRootApi(

// Apply overrides
json = applyOverrides(descriptor, json, config.overrides);
const isCurrent = getApiIsCurrent(json);

// Try to read the specification file for this API
let specImported = false;
const specResult = await store.readContent(config.sourceDir, descriptor, 'specification');
const includeSpecification = options?.includeSpecification ?? true;
const specResult = includeSpecification
? await store.readContent(config.sourceDir, descriptor, 'specification')
: undefined;
if (specResult) {
const properties = json.properties as Record<string, unknown> | undefined;
const apiType = properties?.type as string | undefined;
Expand Down Expand Up @@ -180,9 +205,26 @@ async function publishRootApi(
status: 'success',
action: 'put',
specImported,
isCurrent,
};
}

async function alignActiveRevisionWithSource(
client: IApimClient,
store: IArtifactStore,
context: ApimServiceContext,
descriptor: ResourceDescriptor,
config: PublishConfig
): Promise<RootApiResult & ResourcePublishResult> {
logger.debug(
`Source marks "${getNamePart(descriptor.nameParts, 0)}" as current; re-applying root metadata to align active revision`
);

return publishRootApi(client, store, context, descriptor, config, {
includeSpecification: false,
});
}

/**
* Find and publish API revisions in numeric order
*/
Expand All @@ -192,7 +234,7 @@ async function publishApiRevisions(
context: ApimServiceContext,
apiDescriptor: ResourceDescriptor,
config: PublishConfig
): Promise<void> {
): Promise<number> {
// List all resources from store
const allDescriptors = await store.listResources(config.sourceDir);

Expand All @@ -214,6 +256,8 @@ async function publishApiRevisions(
for (const revDescriptor of sortedRevisions) {
await publishResource(client, store, context, revDescriptor, config);
}

return sortedRevisions.length;
}

/**
Expand Down Expand Up @@ -398,3 +442,9 @@ function extractRevisionNumber(apiName: string): number {
const match = /;rev=(\d+)/.exec(apiName);
return match ? parseInt(match[1], 10) : 0;
}

function getApiIsCurrent(json: Record<string, unknown>): boolean | undefined {
const properties = json.properties as Record<string, unknown> | undefined;
const isCurrent = properties?.isCurrent;
return typeof isCurrent === 'boolean' ? isCurrent : undefined;
}
18 changes: 7 additions & 11 deletions src/services/extract-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -553,18 +553,14 @@ async function extractWorkspaceResources(
filter: FilterConfig | undefined,
result: ExtractionResult
): Promise<void> {
try {
const wsResults = await extractWorkspaces(
client, store, context, outputDir, filter
);
const wsResults = await extractWorkspaces(
client, store, context, outputDir, filter
);

result.workspaceResults = wsResults;
result.workspaceResults = wsResults;

for (const ws of wsResults) {
result.totalExtracted += ws.resourceCount;
result.totalErrors += ws.errorCount;
}
} catch (error) {
logger.warn(`Workspace extraction failed: ${(error as Error).message}`);
for (const ws of wsResults) {
result.totalExtracted += ws.resourceCount;
result.totalErrors += ws.errorCount;
}
}
Loading
Loading