[Utilities] Introduced `Set-Module` script #4067

AlexanderSehr · 2023-10-09T21:42:31Z

Description

Introduced Set-Module script to ease the update of module files & readmes
Updated and moved Set-ModuleReadMe script to pipeline helper location - along with several other helper functions
Updated path reference of diverse utilities to cope with the updated references

TODO:

Pipeline references

Pipeline

Type of Change

Bugfix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Update to documentation

Module content to move

Custom content should be moved into a ## Notes section at the end of the readMe. This should be done before regenerating all readmes as we'd otherwise have a wild mix of old content, new content & deleted content.

This can also be done separate to this PR, but it has to happen before. We can use the below list for tracking which modules with custom content are already done.

aad\domain-service
- '## Considerations'
- '### Create self-signed certificate for secure LDAP'
api-management\service
- '### Parameter Usage: apiManagementServicePolicy'
- '## Considerations'
api-management\service\authorization-server
- '### Resource dependency'
api-management\service\backend
- '### Resource dependency'
- '### Parameter Usage: Credentials'
- '### Parameter Usage: credentials'
- '### Parameter Usage: tls'
api-management\service\named-value
- '### Parameter Usage: keyVault'
authorization\policy-assignment
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '### Parameter Usage: resourceGroupName'
- '## Module Usage Guidance'
authorization\policy-definition
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '## Module Usage Guidance'
authorization\policy-exemption
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '### Parameter Usage: resourceGroupName'
- '### Parameter Usage: resourceSelectors'
- '## Module Usage Guidance'
- '## Considerations'
authorization\policy-set-definition
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '## Module Usage Guidance'
- '## Considerations'
authorization\role-assignment
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '### Parameter Usage: resourceGroupName'
- '## Module Usage Guidance'
- '## Considerations'
authorization\role-definition
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '### Parameter Usage: resourceGroupName'
- '## Module Usage Guidance'
- '## Considerations'
automation\automation-account
- '### Parameter Usage: encryption'
automation\automation-account\software-update-configuration
- '### Parameter Usage: scopeByTags'
- '### Parameter Usage: monthlyOccurrences'
automation\automation-account\variable
- '### Parameter Usage: value'
cache\redis
- '### Parameter Usage: redisConfiguration'
cache\redis-enterprise\database
- '### Parameter Usage: modules'
cognitive-services\account
- '### Parameter Usage: encryption'
- '### Parameter Usage: networkAcls'
- '## Considerations'
compute\gallery\application
- '### Parameter Usage: customActions'
compute\virtual-machine
- '### Parameter Usage: imageReference'
- '#### Marketplace images'
- '#### Custom images'
- '### Parameter Usage: plan'
- '### Parameter Usage: osDisk'
- '### Parameter Usage: dataDisks'
- '### Parameter Usage: nicConfigurations'
- '### Parameter Usage: configurationProfileAssignments'
- '### Parameter Usage: extensionDomainJoinConfig'
- '### Parameter Usage: extensionAntiMalwareConfig'
- '### Parameter Usage: extensionAzureDiskEncryptionConfig'
- '### Parameter Usage: extensionDSCConfig'
- '### Parameter Usage: extensionCustomScriptConfig'
- '### Parameter Usage: extensionCustomScriptProtectedSetting'
- '## Considerations'
compute\virtual-machine-scale-set
- '### Resource dependency'
- '#### Marketplace images'
- '#### Custom images'
- '### Parameter Usage: plan'
- '### Parameter Usage: osDisk'
- '### Parameter Usage: dataDisks'
- '### Parameter Usage: nicConfigurations'
- '### Parameter Usage: extensionDomainJoinConfig'
- '### Parameter Usage: extensionNetworkWatcherAgentConfig'
- '### Parameter Usage: extensionAntiMalwareConfig'
- '### Parameter Usage: extensionAzureDiskEncryptionConfig'
- '### Parameter Usage: extensionCustomScriptConfig'
- '### Parameter Usage: extensionDSCConfig'
container-instance\container-group
- '### Container groups in Azure Container Instances'
- '### Parameter Usage: imageRegistryCredentials'
- '### Parameter Usage: autoGeneratedDomainNameLabelScope'
- '### Parameter Usage: volumes'
container-registry\registry
- '### Parameter Usage: imageRegistryCredentials'
container-service\managed-cluster
- '### Parameter Usage: primaryAgentPoolProfile'
- '### Parameter Usage: httpProxyConfig'
data-factory\factory
- '### Parameter Usage: managedPrivateEndpoints'
data-factory\factory\integration-runtime
- '### Parameter Usage: typeProperties'
data-factory\factory\managed-virtual-network
- '### Parameter Usage: managedPrivateEndpoints'
data-protection\backup-vault
- '### Parameter Usage: backupPolicies'
data-protection\backup-vault\backup-policy
- '### Parameter Usage: properties'
databricks\workspace
- '### Parameter Usage: customPublicSubnetName and customPrivateSubnetName'
- '### Parameter Usage: parameters'
db-for-postgre-sql\flexible-server
- '### Parameter Usage: firewallRules'
- '### Parameter Usage: configurations'
- '### Parameter Usage: databases'
desktop-virtualization\host-pool
- '### Parameter Usage: vmTemplate'
- '### Parameter Usage: customRdpProperty'
desktop-virtualization\scaling-plan
- '### Parameter Usage: schedules'
digital-twins\digital-twins-instance
- '### Parameter Usage: eventhubEndpoint'
- '### Parameter Usage: eventGridEndpoint'
- '### Parameter Usage: serviceBusEndpoint'
document-db\database-account
- '### Parameter Usage: locations'
- '### Parameter Usage: sqlDatabases'
- '### Parameter Usage: mongodbDatabases'
- '### Parameter Usage: gremlinDatabases'
document-db\database-account\gremlin-database
- '### Parameter Usage: graphs'
document-db\database-account\gremlin-database\graph
- '### Parameter Usage: partitionKeyPaths, uniqueKeyPaths'
document-db\database-account\mongodb-database
- '### Parameter Usage: collections'
document-db\database-account\mongodb-database\collection
- '### Parameter Usage: indexes'
- '### Parameter Usage: shardKey'
document-db\database-account\sql-database\container
- '### Parameter Usage: indexingPolicy'
event-grid\system-topic
- '### Parameter Usage: eventSubscriptions'
event-grid\topic
- '### Parameter Usage: eventSubscriptions'
event-hub\namespace\network-rule-set
- '### Parameter Usage: <virtualNetworkRules>'
- '### Parameter Usage: <ipRules>'
healthcare-apis\workspace
- '### Parameter Usage: fhirservices'
- '### Parameter Usage: dicomservices'
- '### Parameter Usage: iotconnectors'
healthcare-apis\workspace\fhirservice
- '### Parameter Usage: acrOciArtifacts'
healthcare-apis\workspace\iotconnector
- '### Parameter Usage: deviceMapping'
- '### Parameter Usage: destinationMapping'
healthcare-apis\workspace\iotconnector\fhirdestination
- '### Parameter Usage: destinationMapping'
insights\action-group
- '### Parameter Usage: receivers'
- '### Additional notes on parameters'
insights\activity-log-alert
- '### Parameter Usage: actions'
- '### Parameter Usage: conditions'
insights\metric-alert
- '### Parameter Usage: actions'
- '### Parameter Usage: criteria'
- '### Additional notes on parameters'
key-vault\vault
- '## Usage examples'
- '### Example 1: Accesspolicies'
- '### Example 2: Common'
- '### Example 3: Min'
- '### Example 4: Pe'
- '### Parameter: accessPolicies'
- '### Parameter: createMode'
- '### Parameter: diagnosticEventHubAuthorizationRuleId'
- '### Parameter: diagnosticEventHubName'
- '### Parameter: diagnosticLogCategoriesToEnable'
- '### Parameter: diagnosticMetricsToEnable'
- '### Parameter: diagnosticSettingsName'
- '### Parameter: diagnosticStorageAccountId'
- '### Parameter: diagnosticWorkspaceId'
- '### Parameter: enableDefaultTelemetry'
- '### Parameter: enablePurgeProtection'
- '### Parameter: enableRbacAuthorization'
- '### Parameter: enableSoftDelete'
- '### Parameter: enableVaultForDeployment'
- '### Parameter: enableVaultForDiskEncryption'
- '### Parameter: enableVaultForTemplateDeployment'
- '### Parameter: keys'
- '### Parameter: location'
- '### Parameter: lock'
- '### Parameter: name'
- '### Parameter: networkAcls'
- '### Parameter: privateEndpoints'
- '### Parameter: publicNetworkAccess'
- '### Parameter: roleAssignments'
- '### Parameter: secrets'
- '### Parameter: softDeleteRetentionInDays'
- '### Parameter: tags'
- '### Parameter: vaultSku'
key-vault\vault\access-policy
- '### Parameter: accessPolicies'
- '### Parameter: enableDefaultTelemetry'
- '### Parameter: keyVaultName'
key-vault\vault\key
- '### Parameter: attributesEnabled'
- '### Parameter: attributesExp'
- '### Parameter: attributesNbf'
- '### Parameter: curveName'
- '### Parameter: enableDefaultTelemetry'
- '### Parameter: keyOps'
- '### Parameter: keySize'
- '### Parameter: keyVaultName'
- '### Parameter: kty'
- '### Parameter: name'
- '### Parameter: roleAssignments'
- '### Parameter: rotationPolicy'
- '### Parameter: tags'
key-vault\vault\secret
- '### Parameter: attributesEnabled'
- '### Parameter: attributesExp'
- '### Parameter: attributesNbf'
- '### Parameter: contentType'
- '### Parameter: enableDefaultTelemetry'
- '### Parameter: keyVaultName'
- '### Parameter: name'
- '### Parameter: roleAssignments'
- '### Parameter: tags'
- '### Parameter: value'
kubernetes-configuration\extension
- '## Prerequisites'
kubernetes-configuration\flux-configuration
- '## Prerequisites'
logic\workflow
- '### Parameter Usage <accessControl>AccessControlConfiguration'
- '### Parameter Usage <flow>EndpointsConfiguration'
- '### Parameter Usage workflow*'
machine-learning-services\workspace
- '### Parameter Usage: computes'
machine-learning-services\workspace\compute
- '### Parameter Usage: properties'
maintenance\maintenance-configuration
- '### Parameter Usage: maintenanceWindow'
managed-services\registration-definition
- '### Parameter Usage: authorizations'
- '## Considerations'
- '### Permissions required to create delegations'
- '### Permissions required to remove delegations'
- '#### From customer side'
- '#### From managing tenant side'
- '### Limitations with Lighthouse and resource delegation'
management\management-group
- '## Considerations'
network\azure-firewall
- '### Parameter Usage: additionalPublicIpConfigurations'
- '### Parameter Usage: publicIPAddressObject'
- '## Considerations'
network\bastion-host
- '### Parameter Usage: additionalPublicIpConfigurations'
- '### Parameter Usage: publicIPAddressObject'
network\connection
- '### Parameter Usage: virtualNetworkGateway1'
- '### Parameter Usage: virtualNetworkGateway2'
- '### Parameter Usage: localNetworkGateway2'
- '### Parameter Usage: peer'
- '### Parameter Usage: customIPSecPolicy'
network\dns-resolver
- '### Parameter Usage: inboundEndpoints'
- '### Parameter Usage: outboundEndpoints'
network\load-balancer
- '### Parameter Usage: frontendIPConfigurations'
- '### Parameter Usage: backendAddressPools'
- '### Parameter Usage: loadBalancingRules'
- '### Parameter Usage: probes'
network\network-interface
- '### Parameter Usage: ipConfigurations'
network\network-manager
- '### Parameter Usage: <networkManagerScopeAccesses>'
- '### Parameter Usage: <networkManagerScopes>'
- '### Parameter Usage: <networkGroups>'
- '### Parameter Usage: <connectivityConfigurations>'
- '### Parameter Usage: <scopeConnections>'
- '### Parameter Usage: <securityAdminConfigurations>'
- '## Considerations'
network\private-dns-zone\txt
- '### Parameter Usage: txtRecords'
network\private-endpoint
- '### Resource dependency'
- '### Parameter Usage: applicationSecurityGroups'
- '### Parameter Usage: customNetworkInterfaceName'
- '### Parameter Usage: ipConfigurations'
network\private-link-service
- '### Parameter Usage: ipConfigurations'
- '### Parameter Usage: loadBalancerFrontendIpConfigurations'
- '### Parameter Usage: extendedLocation'
- '### Parameter Usage: autoApproval'
- '### Parameter Usage: visibility'
- '### Parameter Usage: enableProxyProtocol'
- '### Parameter Usage: fqdns'
network\route-table
- '### Parameter Usage: routes'
network\trafficmanagerprofile
- '### Parameter Usage: monitorConfig'
- '### Parameter Usage: endpoints'
network\virtual-hub\hub-virtual-network-connection
- '### Parameter Usage: hubVirtualNetworkConnections'
network\virtual-network
- '### Parameter Usage: subnets'
- '### Parameter Usage: virtualNetworkPeerings'
- '### Parameter Usage: addressPrefixes'
- '## Considerations'
network\virtual-network\subnet
- '### Parameter Usage: delegations'
- '### Parameter Usage: serviceEndpoints'
- '## Considerations'
network\virtual-network\virtual-network-peering
- '### Resource dependency'
network\virtual-network-gateway
- '### Parameter Usage: subnets'
network\vpn-gateway
- '### Parameter Usage: bgpSettings'
network\vpn-gateway\vpn-connection
- '### Parameter Usage: routingConfiguration'
network\vpn-site
- '### Parameter Usage o365Policy'
- '### Parameter Usage deviceProperties'
- '### Parameter Usage bgpProperties'
- '### Parameter Usage vpnSiteLinks'
operational-insights\workspace
- '### Parameter Usage: gallerySolutions'
policy-insights\remediation
- '### Parameter Usage: managementGroupId'
- '### Parameter Usage: subscriptionId'
- '### Parameter Usage: resourceGroupName'
- '## Module Usage Guidance'
recovery-services\vault
- '### Parameter Usage: backupStorageConfig'
- '### Parameter Usage: backupPolicies'
- '### Parameter Usage: replicationFabrics'
- '### Parameter Usage: replicationPolicies'
recovery-services\vault\backup-policy
- '### Parameter Usage: backupPolicyProperties'
recovery-services\vault\replication-fabric
- '### Parameter Usage: replicationContainers'
recovery-services\vault\replication-fabric\replication-protection-container
- '### Parameter Usage: replicationContainerMappings'
resources\deployment-script
- '## Considerations'
resources\resource-group
- '## Considerations'
security\azure-security-center
- '### Parameter Usage: securityContactProperties'
service-bus\namespace
- '### Parameter Usage: networkAcl'
service-fabric\cluster
- '### Parameter Usage: notifications'
signal-r-service\web-pub-sub
- '### Parameter Usage: networkAcls'
sql\managed-instance
- '### Deployment prerequisites'
- '#### Networking'
- '#### Azure AD Authentication'
- '### Parameter Usage : userAssignedIdentities'
sql\managed-instance\database
- '### Deployment prerequisites'
sql\server
- '### Parameter Usage: administrators'
storage\storage-account
- '### Parameter Usage: networkAcls'
- '## Considerations'
storage\storage-account\management-policy
- '### Parameter Usage: rules'
synapse\workspace\key
- '## Template references'
virtual-machine-images\image-template
- '### Parameter Usage: imageSource'
- '#### Platform Image'
- '#### Managed Image'
- '#### Shared Image'
- '### Parameter Usage: vmUserAssignedIdentities'
web\hosting-environment
- '### Parameter Usage: clusterSettings'
web\serverfarm
- '### Parameter Usage: sku'
web\site
- '### Parameter Usage: appSettingsKeyValuePairs'
- '### Parameter Usage: authSettingV2Configuration'
- '### Parameter Usage: siteConfig'
web\site\config--appsettings
- '### Parameter Usage: appSettingsKeyValuePairs'
web\site\config--authsettingsv2
- '### Parameter Usage: authSettingV2Configuration'
web\site\slot
- '### Parameter Usage: appSettingsKeyValuePairs'
- '### Parameter Usage: authSettingV2Configuration'
- '### Parameter Usage: siteConfig'
web\site\slot\config--appsettings
- '### Parameter Usage: appSettingsKeyValuePairs'
web\site\slot\config--authsettingsv2
- '### Parameter Usage: authSettingV2Configuration'
web\static-site
- '### Parameter Usage: customDomains'

Script to generate readme

$readMePaths = (Get-ChildItem .\modules\ -Recurse -Filter 'main.bicep' -File).FullName | ForEach-Object { Join-Path (Split-Path $_) 'readme.md' }
$customHeaderFiles = $readMePaths | ForEach-Object {
    $name = (Split-Path $_) -replace 'C:\\dev\\ip\\Azure-ResourceModules\\ResourceModules\\modules\\'
    $contentHeaders = Get-Content $_ | Where-Object {
        $_ -like '#*'
    }
    $customHeaders = $contentHeaders | Where-Object {
        $_ -notLike '# *' -and
        $_ -notin @(
            '## Navigation',
            '## Resource types',
            '## Parameters',
            '### Parameter Usage: `roleAssignments`',
            '### Parameter Usage: `tags`',
            '### Parameter Usage: `privateEndpoints`',
            '### Parameter Usage: `userAssignedIdentities`',
            '## Outputs',
            '## Cross-referenced modules',
            '## Deployment examples'
        )
    }
    return [PSCustomObject]@{
        name          = $name
        customHeaders = $customHeaders
    }
} | Where-Object {
    $_.customHeaders.count -gt 0
}

$res = [System.Collections.ArrayList]@()
$customHeaderFiles | ForEach-Object {
    $res += '- [ ] `{0}`' -f $_.name
    $_.customHeaders | ForEach-Object { $res += "  - '$_'" } }
$res | Set-Clipboard
$res

# $res = [System.Collections.ArrayList]@()
# $res += '| Module | Custom headers | Moved |'
# $res += '| - | - | - |'
# $customHeaderFiles | ForEach-Object {
#     $res += ('| {0} | {1} | |' -f $_.name, (($_.customHeaders.count -gt 0) ? $_.customHeaders -join '<br>' : ''))
# }
# $res | Set-Clipboard
# $res

* Regen readmes 3 * Fixed templates * Regen * Renamed header

* Regen readmes 4 * Fixed templates * Regen * Renamed header

* Regen readmes 6 * Fixed templates * Small update * Regen * Renamed header

* Regen readmes 5 * Fixed templates * Regen * Renamed header

* Tested logic & added depth param * Added finally block * Enabled cancel * ReadMe fallback * Added docs * Added silent continue on finally for non-cancel * Update utilities/tools/Set-Module.ps1 Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com> --------- Co-authored-by: Ahmad Abdalla <28486158+ahmadabdalla@users.noreply.github.com>

ahmadabdalla · 2023-10-17T00:24:31Z

@AlexanderSehr I ran the branch on some pipelines to test things:

They're all failing on the Set readme tests :/

ahmadabdalla

Approved! 🚀

AlexanderSehr added 6 commits October 9, 2023 22:12

Moved diverse scripts around and updated them to latest AVM

3582625

Moved further scripts

b94f9d6

Small fix

dd7094f

Further fixes

fc87bd1

Several improvements and fixes

db2d93d

Private registry specific pdates

29d9df1

AlexanderSehr self-assigned this Oct 9, 2023

AlexanderSehr added enhancement New feature or request [cat] utilities category: utilities labels Oct 9, 2023

AlexanderSehr mentioned this pull request Oct 9, 2023

Align in-module folder structure to AVM #4057

Open

5 tasks

AlexanderSehr and others added 20 commits October 9, 2023 23:44

Updated docs

a04c082

Updated path

cace72f

Renamed function

44d7352

Ran generation for KVLT to enable testing of pipeline

f8b5d47

Updated & reduced tests where it made sense

9fb34ab

Fixed pester tests

e7fc051

Moved cross-ref script back to tools

2b1325e

Moved local-ref script back to pipes

2b92886

Fixed ref

2f3430a

Fixed ref

c2911e3

Fixed ref

ee34f88

Fixed ref

cd71f2c

Push updated Readme file(s)

ffa5656

Rollback test changes

8a6200c

Updated test file overview markdown via scrip

e9d1c12

Roll KVLT back post updates

332b362

Updated AAD considerations

2b07459

Removed default headers

3e52a08

Update to latest

de7850f

Enabled notes

92e6c21

AlexanderSehr and others added 10 commits October 16, 2023 21:10

Regenerated modules 121-180 (#4086)

047a1a7

* Regen readmes 3 * Fixed templates * Regen * Renamed header

Regenerated modules 181-240 (#4087)

7c34c7c

* Regen readmes 4 * Fixed templates * Regen * Renamed header

Regenerated modules 300-366 (#4089)

e59098e

* Regen readmes 6 * Fixed templates * Small update * Regen * Renamed header

Regenerated modules 241-300 (#4088)

93bcd42

* Regen readmes 5 * Fixed templates * Regen * Renamed header

Updated readme script

2864324

Updated usage example description

63dc148

Merge branch 'main' into users/alsehr/readmeAVM

9ba9712

Cross ref fix

1cebcd8

Update to latest

6e45890

AlexanderSehr marked this pull request as ready for review October 16, 2023 23:42

AlexanderSehr requested a review from a team as a code owner October 16, 2023 23:42

AlexanderSehr enabled auto-merge (squash) October 16, 2023 23:42

AlexanderSehr requested a review from ahmadabdalla October 16, 2023 23:42

Added verbosity

bfadfd0

AlexanderSehr marked this pull request as draft October 17, 2023 09:48

auto-merge was automatically disabled October 17, 2023 09:48
Pull request was converted to draft

AlexanderSehr added 7 commits October 17, 2023 11:59

Added verbosity

f25a57c

Update to latest

a633b62

Adjusted syntax

ee75550

Adjusted syntax

4a27ab6

Adjusted verbosity

f49edd8

Added force

428eb0c

Cleanup

b29f71a

AlexanderSehr marked this pull request as ready for review October 17, 2023 10:47

AlexanderSehr enabled auto-merge (squash) October 17, 2023 10:47

ahmadabdalla approved these changes Oct 17, 2023

View reviewed changes

AlexanderSehr merged commit 5b43f91 into main Oct 17, 2023
36 of 40 checks passed

AlexanderSehr deleted the users/alsehr/readmeAVM branch October 17, 2023 11:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Utilities] Introduced `Set-Module` script #4067

[Utilities] Introduced `Set-Module` script #4067

AlexanderSehr commented Oct 9, 2023 •

edited

Loading

ahmadabdalla commented Oct 17, 2023

ahmadabdalla left a comment

[Utilities] Introduced Set-Module script #4067

[Utilities] Introduced Set-Module script #4067

Conversation

AlexanderSehr commented Oct 9, 2023 • edited Loading

Description

Pipeline references

Type of Change

Module content to move

ahmadabdalla commented Oct 17, 2023

ahmadabdalla left a comment

Choose a reason for hiding this comment

[Utilities] Introduced `Set-Module` script #4067

[Utilities] Introduced `Set-Module` script #4067

AlexanderSehr commented Oct 9, 2023 •

edited

Loading