Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement RFC 002: Atom Nightly Releases #17538

Merged
merged 30 commits into from Jul 11, 2018
Merged
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
34e37f3
Enable automated nightly Atom releases
daviwil Jun 19, 2018
882648c
Replace app icons
simurai Jun 22, 2018
9388a78
Simplify channel name detection
daviwil Jun 23, 2018
a5a8727
Upload release assets to S3 in script/publish-release
daviwil Jun 23, 2018
ed48f2d
Generate nightly build version from package.json
daviwil Jun 26, 2018
9669a9b
Update app title and file paths to use channel name
daviwil Jun 26, 2018
d54bb62
Update atom.sh to support the nightly channel
daviwil Jun 26, 2018
691d5da
Command installer should use known channel names
daviwil Jun 27, 2018
bd5d258
Detect any channel name in AtomEnvironment.getReleaseChannel
daviwil Jun 27, 2018
ff0cd4c
Use atom.io update endpoint for Windows installer builds
daviwil Jun 28, 2018
067bd1a
Temporarily disable Atom tests on VSTS
daviwil Jun 19, 2018
8e3d1f4
Use Node.js 8.9.3 in VSTS builds
daviwil Jun 25, 2018
cd83486
Fix isReleasedVersion check in AtomEnvironment
daviwil Jun 28, 2018
0d6b5d9
Fix CommandInstaller tests
daviwil Jun 29, 2018
0c364d0
Don't change Atom app name on dev channel
daviwil Jun 29, 2018
ec2abbb
Possible fix for FileRecoveryService test on Linux
daviwil Jun 30, 2018
5634a64
Re-enable Windows and Linux tests on VSTS
daviwil Jun 30, 2018
9858800
Mock fs.createWriteStream to fix FileRecoveryService test
daviwil Jun 30, 2018
7750b8c
Re-enable macOS VSTS CI tests
daviwil Jul 2, 2018
5917800
Add CI_PROVIDER env variable to test tasks
daviwil Jul 6, 2018
4527a44
Merge branch 'master' into dw-nightly-releases
daviwil Jul 6, 2018
f1c7ddf
:fire: script/lib/create-github-release.js
daviwil Jul 6, 2018
e91030f
Set timeout of 180 minutes on nightly build
daviwil Jul 7, 2018
0d0db81
Remove queue setting in nightly build
daviwil Jul 7, 2018
49b9b9e
Write error to stderr if publishing release fails
daviwil Jul 9, 2018
7b711cd
Add documentation for Atom Nightly build process
daviwil Jul 9, 2018
fa326a0
Minor tweaks to build documentation
daviwil Jul 9, 2018
bea3267
Simplify atom-x64 nupkg path generation
daviwil Jul 9, 2018
3046f68
Try possible fix for broken GitHub package tests
daviwil Jul 10, 2018
3109958
Revert "Try possible fix for broken GitHub package tests"
daviwil Jul 11, 2018
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.
+341 −22
Diff settings

Always

Just for now

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
@@ -104,8 +104,10 @@ if (!argv.generateApiDocs) {
}
if (argv.createWindowsInstaller) {
return createWindowsInstaller(packagedAppPath)
.then(() => argv.codeSign && codeSignOnWindows([ path.join(CONFIG.buildOutputPath, 'AtomSetup.exe') ]))
.then(() => packagedAppPath)
.then((installerPath) => {
argv.codeSign && codeSignOnWindows([installerPath])
return packagedAppPath
})
} else {
console.log('Skipping creating installer. Specify the --create-windows-installer option to create a Squirrel-based Windows installer.'.gray)
}
@@ -5,6 +5,7 @@

const fs = require('fs')
const path = require('path')
const spawnSync = require('./lib/spawn-sync')

const repositoryRootPath = path.resolve(__dirname, '..')
const apmRootPath = path.join(repositoryRootPath, 'apm')
@@ -20,11 +21,13 @@ const atomHomeDirPath = process.env.ATOM_HOME || path.join(homeDirPath, '.atom')
const appMetadata = require(path.join(repositoryRootPath, 'package.json'))
const apmMetadata = require(path.join(apmRootPath, 'package.json'))
const channel = getChannel()
const computedAppVersion = computeAppVersion(appMetadata.version, channel)

module.exports = {
appMetadata,
apmMetadata,
channel,
computedAppVersion,
repositoryRootPath,
apmRootPath,
scriptRootPath,
@@ -41,7 +44,9 @@ module.exports = {
}

function getChannel () {
if (appMetadata.version.match(/dev/)) {
if (process.env.BUILD_DEFINITIONNAME === 'Atom Nightly') {

This comment has been minimized.

Copy link
@daviwil

daviwil Jun 19, 2018

Author Member

This is a VSTS environment variable but I'm planning to change it to something specific to Atom

return 'nightly'
} else if (appMetadata.version.match(/dev/)) {
return 'dev'
} else if (appMetadata.version.match(/beta/)) {
return 'beta'
@@ -50,6 +55,17 @@ function getChannel () {
}
}

function computeAppVersion (version, channel) {
if (channel === 'dev') {
const result = spawnSync('git', ['rev-parse', '--short', 'HEAD'], {cwd: repositoryRootPath})
const commitHash = result.stdout.toString().trim()
version += '-' + commitHash
} else if (channel === 'nightly') {
version = process.env.BUILD_BUILDNUMBER

This comment has been minimized.

Copy link
@daviwil

daviwil Jun 19, 2018

Author Member

I'll also change this env variable to something that isn't specific to VSTS

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Jun 20, 2018

Member

It looks like this uses the build number as Atom's version number. If that's the current behavior, how would you feel about changing it to append the build number to the version number (e.g., "1.30.0-nightly-42")?

This comment has been minimized.

Copy link
@daviwil

daviwil Jul 11, 2018

Author Member

This was actually the case but it wasn't super clear since I was using VSTS' environment variable :) At this point in the code, I had configured the build to use the full 1.30.0-nightly42 version as the "build number". I've since changed it to use a more Atom-specific environment variable that gets set in the VSTS YAML config.

}
return version
}

function getApmBinPath () {
const apmBinName = process.platform === 'win32' ? 'apm.cmd' : 'apm'
return path.join(apmRootPath, 'node_modules', 'atom-package-manager', 'bin', apmBinName)
@@ -16,6 +16,36 @@ module.exports = function (packagedAppPath) {
downloadFileFromGithub(process.env.ATOM_MAC_CODE_SIGNING_CERT_DOWNLOAD_URL, certPath)
}
try {
console.log(`Ensuring keychain ${process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN} exists`)
try {
spawnSync('security', [
'show-keychain-info',
process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN
], {stdio: 'inherit'})
} catch (err) {
console.log(`Creating keychain ${process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN}`)
// The keychain doesn't exist, try to create it
spawnSync('security', [
'create-keychain',
'-p', process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN_PASSWORD,
process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN
], {stdio: 'inherit'})

// List the keychain to "activate" it. Somehow this seems
// to be needed otherwise the signing operation fails
spawnSync('security', [
'list-keychains',
'-s', process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN
], {stdio: 'inherit'})

// Make sure it doesn't time out before we use it
spawnSync('security', [
'set-keychain-settings',
'-t', '3600',
'-u', process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN
], {stdio: 'inherit'})
}

console.log(`Unlocking keychain ${process.env.ATOM_MAC_CODE_SIGNING_KEYCHAIN}`)
const unlockArgs = ['unlock-keychain']
// For signing on local workstations, password could be entered interactively
@@ -3,6 +3,7 @@
const fs = require('fs-extra')
const path = require('path')
const spawnSync = require('./spawn-sync')
const { path7za } = require('7zip-bin')

const CONFIG = require('../config')

@@ -19,7 +20,7 @@ module.exports = function (packagedAppPath) {
function getArchiveName () {
switch (process.platform) {
case 'darwin': return 'atom-mac.zip'
case 'win32': return 'atom-windows.zip'
case 'win32': return `atom-${process.arch === 'x64' ? 'x64-' : ''}windows.zip`
default: return `atom-${getLinuxArchiveArch()}.tar.gz`
}
}
@@ -44,7 +45,7 @@ function compress (inputDirPath, outputArchivePath) {
compressCommand = 'zip'
compressArguments = ['-r', '--symlinks']
} else if (process.platform === 'win32') {
compressCommand = '7z.exe'
compressCommand = path7za
compressArguments = ['a', '-r']
} else {
compressCommand = 'tar'
@@ -0,0 +1,30 @@
'use strict'

This comment has been minimized.

Copy link
@daviwil

daviwil Jun 19, 2018

Author Member

This file wasn't supposed to make it in, will take it back out


const publishRelease = require('publish-release')
const CONFIG = require('../config')

module.exports = function (assets) {
return new Promise(function (resolve, reject) {
console.log(`Uploading assets to GitHub release ${CONFIG.computedAppVersion}`)
publishRelease({
token: process.env.GITHUB_TOKEN,
owner: 'atom',
repo: CONFIG.channel !== 'nightly' ? 'atom' : 'atom-nightly-releases',
name: CONFIG.computedAppVersion,
tag: CONFIG.computedAppVersion,
draft: true,
prerelease: CONFIG.channel !== 'stable',
reuseRelease: true,
reuseDraftOnly: true,
skipIfPublished: true,
assets
}, function (err, release) {
if (err) {
reject(err)
} else {
console.log('Release created successfully: ', release.html_url)
resolve(release)
}
})
})
}
@@ -1,32 +1,45 @@
'use strict'

const electronInstaller = require('electron-winstaller')
const fs = require('fs-extra')
const fs = require('fs')
const glob = require('glob')
const path = require('path')

const CONFIG = require('../config')

module.exports = (packagedAppPath) => {
const archSuffix = process.arch === 'ia32' ? '' : '-' + process.arch
// const archSuffix = process.arch === 'ia32' ? '' : '-' + process.arch

This comment has been minimized.

Copy link
@daviwil

daviwil Jun 19, 2018

Author Member

This and the other commented line below will be uncommented once atom.io changes are merged/deployed.

const options = {
appDirectory: packagedAppPath,
authors: 'GitHub Inc.',
iconUrl: `https://raw.githubusercontent.com/atom/atom/master/resources/app-icons/${CONFIG.channel}/atom.ico`,
loadingGif: path.join(CONFIG.repositoryRootPath, 'resources', 'win', 'loading.gif'),
outputDirectory: CONFIG.buildOutputPath,
noMsi: true,
remoteReleases: `https://atom.io/api/updates${archSuffix}?version=${CONFIG.appMetadata.version}`,
// remoteReleases: `https://atom.io/api/updates${archSuffix}?version=${CONFIG.appMetadata.version}`,
setupExe: `AtomSetup${process.arch === 'x64' ? '-x64' : ''}.exe`,
setupIcon: path.join(CONFIG.repositoryRootPath, 'resources', 'app-icons', CONFIG.channel, 'atom.ico')
}

const cleanUp = () => {
for (let nupkgPath of glob.sync(`${CONFIG.buildOutputPath}/*.nupkg`)) {
const releasesPath = `${CONFIG.buildOutputPath}/RELEASES`
if (process.arch === 'x64' && fs.existsSync(releasesPath)) {
fs.renameSync(releasesPath, `${releasesPath}-x64`)
}

for (let nupkgPath of glob.sync(`${CONFIG.buildOutputPath}/atom-*.nupkg`)) {
if (!nupkgPath.includes(CONFIG.appMetadata.version)) {
console.log(`Deleting downloaded nupkg for previous version at ${nupkgPath} to prevent it from being stored as an artifact`)
fs.removeSync(nupkgPath)
fs.unlinkSync(nupkgPath)
} else {
if (process.arch === 'x64') {
const newNupkgPath = `${CONFIG.buildOutputPath}/atom-x64${path.basename(nupkgPath).slice(4)}`

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Jun 20, 2018

Member

What's the significance of the number 4 on this line? Could that number be extracted to a constant whose name explains the significant of the number?

This comment has been minimized.

Copy link
@daviwil

daviwil Jul 9, 2018

Author Member

4 is the length of the string atom, I'm basically ripping the front part of the original filename atom-1.30.0-nightly00-full.nupkg off so that I can replace it with atom-x64 in the build output path. I've simplified the logic here and added a comment so that the intent is clearer.

fs.renameSync(nupkgPath, newNupkgPath)
}
}
}

return `${CONFIG.buildOutputPath}/${options.setupExe}`
}

console.log(`Creating Windows Installer for ${packagedAppPath}`)
@@ -6,7 +6,6 @@ const fs = require('fs-plus')
const normalizePackageData = require('normalize-package-data')
const path = require('path')
const semver = require('semver')
const spawnSync = require('./spawn-sync')

const CONFIG = require('../config')

@@ -16,7 +15,7 @@ module.exports = function () {
CONFIG.appMetadata._atomMenu = buildPlatformMenuMetadata()
CONFIG.appMetadata._atomKeymaps = buildPlatformKeymapsMetadata()
CONFIG.appMetadata._deprecatedPackages = deprecatedPackagesMetadata
CONFIG.appMetadata.version = computeAppVersion()
CONFIG.appMetadata.version = CONFIG.computedAppVersion
checkDeprecatedPackagesMetadata()
fs.writeFileSync(path.join(CONFIG.intermediateAppPath, 'package.json'), JSON.stringify(CONFIG.appMetadata))
}
@@ -162,13 +161,3 @@ function checkDeprecatedPackagesMetadata () {
}
}
}

function computeAppVersion () {
let version = CONFIG.appMetadata.version
if (CONFIG.channel === 'dev') {
const result = spawnSync('git', ['rev-parse', '--short', 'HEAD'], {cwd: CONFIG.repositoryRootPath})
const commitHash = result.stdout.toString().trim()
version += '-' + commitHash
}
return version
}
@@ -2,6 +2,7 @@
"name": "atom-build-scripts",
"description": "Atom build scripts",
"dependencies": {
"7zip-bin": "^4.0.2",
"async": "2.0.1",
"babel-core": "5.8.38",
"coffeelint": "1.15.7",
@@ -26,6 +27,7 @@
"npm": "5.3.0",
"passwd-user": "2.1.0",
"pegjs": "0.9.0",
"publish-release": "^1.6.0",
"random-seed": "^0.3.0",
"season": "5.3.0",
"semver": "5.3.0",
@@ -0,0 +1,40 @@
#!/usr/bin/env node

'use strict'

const path = require('path')
const glob = require('glob')
const publishRelease = require('publish-release')
const CONFIG = require('./config')

console.log(`Publishing GitHub release ${CONFIG.computedAppVersion}`)

const yargs = require('yargs')
const argv = yargs
.usage('Usage: $0 [options]')
.help('help')
.describe('assets-path', 'Path to the folder where all release assets are stored')
.wrap(yargs.terminalWidth())
.argv

let assetsPath = argv.assetsPath || path.join(CONFIG.repositoryRootPath, 'out')
let assets = glob.sync(path.join(assetsPath, '*(*.exe|*.zip|*.nupkg|*.tar.gz|*.rpm|*.deb|RELEASES*)'))

publishRelease({
token: process.env.GITHUB_TOKEN,
owner: 'atom',
repo: CONFIG.channel !== 'nightly' ? 'atom' : 'atom-nightly-releases',

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Jun 20, 2018

Member

Can you remind me why it makes sense to have a separate repository for the nightly releases?

This comment has been minimized.

Copy link
@daviwil

daviwil Jul 11, 2018

Author Member

Two reasons:

  1. atom/atom's Releases feed will get swamped with nightly releases and it'll be harder to scan through the list to find older stable/beta releases
  2. My goal was to use Electron's new update service for this and they use the Releases feed as the source of truth for updates. Seemed to make more sense to have it be a separate repo for now. Can always change it later!
name: CONFIG.computedAppVersion,
tag: CONFIG.computedAppVersion,
draft: false,
prerelease: CONFIG.channel !== 'stable',
reuseRelease: true,
skipIfPublished: true,
assets
}, function (err, release) {
if (err) {
console.log("An error occurred while publishing the release:\n\n", err)
} else {
console.log("Release published successfully: ", release.html_url)
}
})
@@ -0,0 +1,5 @@
@IF EXIST "%~dp0\node.exe" (
"%~dp0\node.exe" "%~dp0\publish-release" %*
) ELSE (
node "%~dp0\publish-release" %*
)
@@ -0,0 +1,52 @@
phases:
- phase: Linux
queue:
name: Hosted Linux Preview
timeoutInMinutes: 180

steps:
- task: NodeTool@0
inputs:
versionSpec: 8.11.3

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Jun 20, 2018

Member

I think the current builds all use Node 6.9.4 [example]. Can you share the motivation for using Node 8.11.3 here?

I'm thinking that we'd want all builds (including the new nightly builds) using the same Node version.

This comment has been minimized.

Copy link
@daviwil

daviwil Jul 11, 2018

Author Member

Mainly because we now use Node 8 inside of Electron 2.0, so it's helpful to have the same JavaScript language features (async/await, improved perf) inside our build scripts. I started off using Node 6 in the builds and saw a noticable performance increase (~10 minutes saved) by switching to 8. More recently in this PR, I switched to the exact version of node in Electron 2.0, 8.9.3.

This comment has been minimized.

Copy link
@jasonrudolph

jasonrudolph Jul 11, 2018

Member

@daviwil Thanks for the insight. Can I talk you into opening a follow-up PR to replace the existing uses of Node 6 with Node 8?

env: NODE_VERSION=6.9.4 DISPLAY=:99.0 CC=clang CXX=clang++ npm_config_clang=1

NODE_VERSION: 6.9.4

atom/circle.yml

Lines 20 to 21 in 3109958

- nvm install 6.9.4
- nvm use 6.9.4

This comment has been minimized.

Copy link
@daviwil

daviwil Jul 11, 2018

Author Member

Definitely, I'll send that out later today!

displayName: Install Node.js 8.11.3

- script: |
apt-get update
apt-get install -y --no-install-recommends build-essential xvfb clang-3.5 fakeroot git libsecret-1-dev rpm libx11-dev libxkbfile-dev xz-utils xorriso zsync libxss1 libgconf2-4 libgtk-3-0
displayName: Install apt dependencies
- script: |
script/build --create-debian-package --create-rpm-package --compress-artifacts
displayName: Build Atom
- script: script/lint
displayName: Run linter

- script: |
/sbin/start-stop-daemon --start --quiet --pidfile /tmp/custom_xvfb_99.pid --make-pidfile --background --exec /usr/bin/Xvfb -- :99 -ac -screen 0 1280x1024x16
export DISPLAY=':99.0'
Xvfb :99 -screen 0 1024x768x24 > /dev/null 2>&1 &
script/test
env:
CI: true
displayName: Run tests
# This step is necessary in the short term due to a bug in the *NIX
# implementation of the CopyFiles task which scans the entire file
# system structure just to resolve the glob pattern.
- script: rm -rf $(Build.SourcesDirectory)/out/*/
displayName: Delete Intermediate Output

- task: CopyFiles@2
inputs:
sourceFolder: $(Build.SourcesDirectory)/out
contents: '?(*.deb|*.rpm|*.tar.gz)'
targetFolder: $(Build.ArtifactStagingDirectory)
displayName: Stage Artifacts

- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: Binaries
ArtifactType: Container
displayName: Upload Artifacts
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.