Skip to content

feat: add nav-data support to compare-api-responses script #385

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 7, 2025
Merged

feat: add nav-data support to compare-api-responses script #385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion app/utils/productConfig.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,12 +54,13 @@ import semver from 'semver'
* basePaths: paths where content may exist on the website for the product, used for rewrite-internal-links script. Not required for all products
* contentDir: directory where content lives in the product repo, used for migration script and all-docs-paths api route
* dataDir: directory where nav data lives in the product repo, used for migration script
* navDataPath: path used as a prefix for the nav data in the product repo, used for api compare tool
* productSlug: product that is associated with the product repo, used for all-docs-paths api route
* semverCoerce: a function that coerces a version string into a semver version object, used for migration scripts
* versionedDocs: a boolean that indicates whether the product has versioned docs
* websiteDir: directory where all docs content folders live in the product repo, used for migration script
*
* @type Record<string, { assetDir: string, basePaths?: string[], contentDir: string, dataDir: string, productSlug: string, semverCoerce: Function, versionedDocs: boolean, websiteDir: string }>
* @type Record<string, { assetDir: string, basePaths?: string[], contentDir: string, dataDir: string, navDataPath: string, productSlug: string, semverCoerce: Function, versionedDocs: boolean, websiteDir: string }>
*/

export const PRODUCT_CONFIG = {
Expand Down Expand Up @@ -251,6 +252,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['cdktf'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'cdktf',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand All @@ -270,6 +272,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['cloud-docs/agents'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'cloud-docs-agents',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand Down Expand Up @@ -313,6 +316,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['plugin/framework'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'plugin-framework',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand All @@ -326,6 +330,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['plugin/log'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'plugin-log',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand All @@ -347,6 +352,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['plugin/mux'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'plugin-mux',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand All @@ -368,6 +374,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['plugin/sdkv2'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'plugin-sdkv2',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand All @@ -388,6 +395,7 @@ export const PRODUCT_CONFIG = {
basePaths: ['plugin/testing'],
contentDir: 'docs',
dataDir: 'data',
navDataPath: 'plugin-testing',
productSlug: 'terraform',
semverCoerce: semver.coerce,
versionedDocs: true,
Expand Down
197 changes: 109 additions & 88 deletions scripts/compare-api-responses/index.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,40 +90,9 @@ function saveTestOutputIfSelected(outputString, newApiURL) {
}
}

async function fetchApiTypeVersionAndNav(options, product, versionMetadata) {
Copy link
Contributor Author

@LeahMarieBush LeahMarieBush May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since I split up version-metadata and nav-data options and nav-data uses a different response value than what is returned here, it made sense to me to pull this out and put it within each case

let newApiURL
let oldApiURL

if (options.api === 'version-metadata') {
newApiURL = `${options.newApiUrl}/api/content/${product}/version-metadata`
oldApiURL = `${options.oldApiUrl}/api/content/${product}/version-metadata?partial=true`
} else if (options.api === 'nav-data') {
newApiURL = `${options.newApiUrl}/api/content/${product}/nav-data/${versionMetadata.version}`
oldApiURL = `${options.oldApiUrl}/api/content/${product}/nav-data/${versionMetadata.version}`
}

const newApiResponse = await fetch(newApiURL)
const oldApiResponse = await fetch(oldApiURL)

const newApiData = await newApiResponse.json()
const oldApiData = await oldApiResponse.json()

const newApiDataStrings = JSON.stringify(
sortObjectByKeys(newApiData.result),
null,
2,
).split('\n')
const oldApiDataStrings = JSON.stringify(
sortObjectByKeys(oldApiData.result),
null,
2,
).split('\n')

return { newApiDataStrings, oldApiDataStrings, newApiURL }
}

const testsPassed = []
const testsFailed = []
const totalTests = []
Copy link
Contributor Author

@LeahMarieBush LeahMarieBush May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added totalTests here since options.numOfTests may not be defined for every test run. This way the test results do not print 'Tests passed: 14 of undefined'

for (const [product, versions] of Object.entries(versionMetadata)) {
if (options.product && options.product !== product) {
continue
Expand All @@ -134,9 +103,26 @@ for (const [product, versions] of Object.entries(versionMetadata)) {
continue
}

if (options.api === 'version-metadata' || options.api === 'nav-data') {
const { newApiDataStrings, oldApiDataStrings, newApiURL } =
await fetchApiTypeVersionAndNav(options, product, versionMetadata)
if (options.api === 'version-metadata') {
const newApiURL = `${options.newApiUrl}/api/content/${product}/version-metadata`
const oldApiURL = `${options.oldApiUrl}/api/content/${product}/version-metadata?partial=true`

const newApiResponse = await fetch(newApiURL)
const oldApiResponse = await fetch(oldApiURL)

const newApiData = await newApiResponse.json()
const oldApiData = await oldApiResponse.json()

const newApiDataStrings = JSON.stringify(
sortObjectByKeys(newApiData.result),
null,
2,
).split('\n')
const oldApiDataStrings = JSON.stringify(
sortObjectByKeys(oldApiData.result),
null,
2,
).split('\n')

const diffOptions = {
contextLines: 1,
Expand All @@ -154,6 +140,39 @@ for (const [product, versions] of Object.entries(versionMetadata)) {

saveTestOutputIfSelected(outputString, newApiURL)
break
} else if (options.api === 'nav-data') {
const newApiURL = `${options.newApiUrl}/api/content/${product}/nav-data/${versionMetadata.version}/${contentDirMap[product].navDataPath}`
const oldApiURL = `${options.oldApiUrl}/api/content/${product}/nav-data/${versionMetadata.version}/${contentDirMap[product].navDataPath}`

const newApiResponse = await fetch(newApiURL)
const oldApiResponse = await fetch(oldApiURL)

const newApiData = await newApiResponse.json()
const oldApiData = await oldApiResponse.json()

const newDataToCompare = newApiData.result?.navData
const oldDataToCompare = oldApiData.result?.navData

const diffOptions = {
contextLines: 1,
expand: false,
}

const difference = diff(oldDataToCompare, newDataToCompare, diffOptions)

const outputString = `Testing API URL:\n${newApiURL}\n${difference}`
console.log(outputString)

if (difference.includes('Compared values have no visual difference.')) {
testsPassed.push(newApiURL)
console.log('✅ No visual difference found.\n')
} else {
testsFailed.push(newApiURL)
console.log(`${difference}\n`)
}

totalTests.push(newApiURL)
saveTestOutputIfSelected(outputString, newApiURL)
} else if (options.api === 'content') {
const productContentDir = contentDirMap[product].contentDir
const isVersionedProduct = contentDirMap[product].versionedDocs
Expand Down Expand Up @@ -246,73 +265,75 @@ for (const [product, versions] of Object.entries(versionMetadata)) {
console.log(`${difference}\n`)
}

totalTests.push(i + 1)
saveTestOutputIfSelected(outputString, newApiURL)
}
} else if (options.api === 'content-versions') {
Copy link
Contributor Author

@LeahMarieBush LeahMarieBush May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'content-versions' got moved out of the for loops for versionmetadata and product because it does not use either thing in the loops. Since I removed the break statements for the loops, this would get run a bunch of times with the same output.

const newApiURL = new URL(options.newApiUrl)
const oldApiURL = new URL(options.oldApiUrl)

let newApiResponse, oldApiResponse
try {
newApiResponse = await fetch(newApiURL)
oldApiResponse = await fetch(oldApiURL)

if (!newApiResponse.ok) {
console.log(
`Error fetching API response:\n${newApiURL}\n${newApiResponse.statusText}`,
)
continue
}
if (!oldApiResponse.ok) {
console.log(
`Error fetching API response:\n${oldApiURL}\n${oldApiResponse.statusText}`,
)
continue
}
} catch (error) {
console.log(`Error fetching API response\n${error}`)
continue
}
}
}
}

let newApiData, oldApiData
try {
newApiData = await newApiResponse.json()
oldApiData = await oldApiResponse.json()
} catch (error) {
console.log(`Error decoding JSON\n${error}`)
continue
}
if (options.api === 'content-versions') {
const newApiURL = new URL(options.newApiUrl)
const oldApiURL = new URL(options.oldApiUrl)

let newApiResponse, oldApiResponse
try {
newApiResponse = await fetch(newApiURL)
oldApiResponse = await fetch(oldApiURL)

// sort the versions to normalize the responses because the APIs return content versions in different orders
const newSortedVersions = newApiData.versions.sort()
const oldSortedVersions = oldApiData.versions.sort()
if (!newApiResponse.ok) {
console.log(
`Error fetching API response:\n${newApiURL}\n${newApiResponse.statusText}`,
)
}
if (!oldApiResponse.ok) {
console.log(
`Error fetching API response:\n${oldApiURL}\n${oldApiResponse.statusText}`,
)
}
} catch (error) {
console.log(`Error fetching API response\n${error}`)
}

const difference = diff(newSortedVersions, oldSortedVersions)
let newApiData, oldApiData
try {
newApiData = await newApiResponse.json()
oldApiData = await oldApiResponse.json()
} catch (error) {
console.log(`Error decoding JSON\n${error}`)
}

const outputString = `Testing API URL\n${difference}`
console.log(outputString)
// sort the versions to normalize the responses because the APIs return content versions in different orders
const newSortedVersions = newApiData.versions.sort()
const oldSortedVersions = oldApiData.versions.sort()

if (difference.includes('Compared values have no visual difference.')) {
testsPassed.push(newApiURL)
console.log('✅ No visual difference found.\n')
} else {
testsFailed.push(newApiURL)
console.log(`${difference}\n`)
}
const difference = diff(newSortedVersions, oldSortedVersions)

saveTestOutputIfSelected(outputString, newApiURL)
}
const outputString = `Testing API URL\n${difference}`
console.log(outputString)

break
if (difference.includes('Compared values have no visual difference.')) {
testsPassed.push(newApiURL)
console.log('✅ No visual difference found.\n')
} else {
testsFailed.push(newApiURL)
console.log(`${difference}\n`)
}

break
Comment on lines -306 to -309
Copy link
Contributor Author

@LeahMarieBush LeahMarieBush May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These break statements are removed because they are causing the tests to only run on the first version of the first product. Now, all products and versions get tests run on them

totalTests.push(newApiURL)
saveTestOutputIfSelected(outputString, newApiURL)
}

if (options.api === 'content') {
if (
options.api === 'content' ||
options.api === 'nav-data' ||
options.api === 'content-versions'
) {
console.log(
`Tests passed: ${testsPassed.length} of ${options.numOfTests} ${
testsPassed.length === options.numOfTests ? '🎉' : ''
`Tests passed: ${testsPassed.length} of ${options.numOfTests ?? totalTests.length} ${
testsPassed.length === (options.numOfTests ?? totalTests.length)
Comment on lines +333 to +334
Copy link
Contributor Author

@LeahMarieBush LeahMarieBush May 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to comment above about totalTests. The options.numOfTests is not always defined so we need a fallback value

? '🎉'
: ''
}`,
)
}
Loading