Skip to content

Fetch JSON spec from Artifacts Snapshot API #2949

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 1, 2024
Merged

Fetch JSON spec from Artifacts Snapshot API #2949

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
90 changes: 20 additions & 70 deletions .github/download-artifacts/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,23 +40,21 @@ const downloadedSpec = join(esFolder, 'rest-api-spec', 'api')
const specFolder = join(__dirname, '..', '..', 'specification', '_json_spec')

async function downloadArtifacts (opts) {
if (typeof opts.version !== 'string' && typeof opts.branch !== 'string') {
throw new Error('Missing version or branch')
if (typeof opts.branch !== 'string') {
throw new Error('Missing branch')
}

core.info('Checking out spec and test')
core.info('Resolving artifact URL')

core.info('Resolving version')
let resolved
try {
resolved = await resolve(opts.version || fromBranch(opts.branch), opts.hash)
resolved = await resolve(opts.branch)
} catch (err) {
core.error(err.message)
process.exit(1)
}

opts.version = resolved.version
core.info(`Resolved version ${opts.version}`)
core.info(`Resolved artifact URL for ${resolved.commit_url}`)

core.info('Cleanup')
await rm(esFolder)
Expand Down Expand Up @@ -96,74 +94,26 @@ async function downloadArtifacts (opts) {
core.info('Done')
}

async function resolve (version, hash) {
if (version === 'latest') {
const response = await fetch('https://artifacts-api.elastic.co/v1/versions')
if (!response.ok) {
throw new Error(`unexpected response ${response.statusText}`)
}
const { versions } = await response.json()
version = versions.pop()
}

core.info(`Resolving version ${version}`)
const response = await fetch(`https://artifacts-api.elastic.co/v1/versions/${version}`)
async function resolve (branch) {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is largely copied over from clients-flight-recorder.

const url = `https://artifacts-snapshot.elastic.co/elasticsearch/latest/${branch}.json`
const response = await fetch(url)
if (!response.ok) {
throw new Error(`unexpected response ${response.statusText}`)
throw new Error(`Unexpected response. Invalid version? ${url}: ${response.statusText}`)
}

const data = await response.json()
const esBuilds = data.version.builds
.filter(build => build.projects.elasticsearch != null)
.map(build => {
return {
projects: build.projects.elasticsearch,
buildId: build.build_id,
date: build.start_time,
version: build.version
}
})
.sort((a, b) => {
const dA = new Date(a.date)
const dB = new Date(b.date)
if (dA > dB) return -1
if (dA < dB) return 1
return 0
})

if (hash != null) {
const build = esBuilds.find(build => build.projects.commit_hash === hash)
if (!build) {
throw new Error(`Can't find any build with hash '${hash}'`)
}
const zipKey = Object.keys(build.projects.packages).find(key => key.startsWith('rest-resources-zip-') && key.endsWith('.zip'))
return {
url: build.projects.packages[zipKey].url,
id: build.buildId,
hash: build.projects.commit_hash,
version: build.version
}
}

const lastBuild = esBuilds[0]
const zipKey = Object.keys(lastBuild.projects.packages).find(key => key.startsWith('rest-resources-zip-') && key.endsWith('.zip'))
return {
url: lastBuild.projects.packages[zipKey].url,
id: lastBuild.buildId,
hash: lastBuild.projects.commit_hash,
version: lastBuild.version
let manifest_url = data.manifest_url
const manifestResponse = await fetch(manifest_url)
if (!manifestResponse.ok) {
throw new Error(`Unexpected manifestResponse. ${manifest_url}: ${manifestResponse.statusText}`)
}
}
const manifestData = await manifestResponse.json()
const elasticsearch = manifestData.projects.elasticsearch
const restResourceName = `rest-resources-zip-${manifestData.version}.zip`

function fromBranch (branch) {
if (branch === 'main') {
return 'latest'
} else if (branch === '7.x') {
return '7.x-SNAPSHOT'
} else if ((branch.startsWith('7.') || branch.startsWith('8.')) && !isNaN(Number(branch.split('.')[1]))) {
return `${branch}-SNAPSHOT`
} else {
throw new Error(`Cannot derive version from branch '${branch}'`)
return {
url: elasticsearch.packages[restResourceName].url,
commit_url: elasticsearch.commit_url,
}
}

Expand All @@ -172,7 +122,7 @@ async function main (options) {
}

const options = minimist(process.argv.slice(2), {
string: ['id', 'version', 'hash', 'branch']
string: ['branch']
Comment on lines -175 to +125
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only use --branch now.

})
main(options).catch(t => {
core.error(t)
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/update-rest-api-json.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:

- name: Generate output
run: |
SKIP_VERSION_UPDATE=true make contrib
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This had no effect for some time now.

make contrib

- name: Debug git status
run: |
Expand Down
Loading