Download platform/architecture specific binaries #33

jperl · 2020-01-14T00:21:46Z

Resolves #20
Resolves https://github.com/qawolf/qawolf/issues/316

To Do

Create GitHub action to download & release binaries on v* tags
Create install.js to download binary from release
Use separate binary_release for the version
Clean up install.js
Use non-deprecated github action
Alert user in install.js if they have an invalid platform
Compress to reduce download size

jperl · 2020-01-14T00:56:21Z

Instead of rely on the package version I will create a separate binary tag / release and include it in the package.json.

Now the binaries github action can be triggered and released separately from the package version.
This makes it possible to run the tests against different binary releases. Otherwise you would not be able to run the tests until after a release.

.github/workflows/release-binaries.yml

.travis.yml

derhuerst · 2020-01-14T16:42:34Z

index.js

@@ -1,24 +1,22 @@
-var os = require('os')
-var path = require('path')
+var os = require("os");


Please revert these changes. They make it really hard to find out later when & how something was changed.

derhuerst · 2020-01-14T16:42:43Z

install.js

+var URL = require("url");
+var pkg = require("./package");
+
+function httpRequest(url, method, response) {


Use a slightly-higher-level lib like simple-get for this. Unfortunately we'll still disrepect other important parts of the HTTP ecosystem, e.g. caching & proxy support.

derhuerst · 2020-01-14T16:48:17Z

Now the binaries github action can be triggered and released separately from the package version.

I think that makes a lot of sense because it decouples ffmpeg releases from ffmpeg-static releases (in both ways).

This makes it possible to run the tests against different binary releases. Otherwise you would not be able to run the tests until after a release.

The tests don't assert the functionality of the ffmpeg binaries anyways, so we don't need that.

- just try on two for now

…riable

jperl · 2020-01-15T18:23:48Z

@derhuerst Sorry for the delay on wrapping this up I am taking a vacation this week and working on this when I can.

The PR should be ready for review now, I made the changes requested. Overall I think this approach ended up being simpler than the multi-package approach.

jperl · 2020-01-15T18:31:11Z

Note: The last commit tests are failing because that commit changes the download url to this repo, and there is not a binary release here yet.

After this PR is merged you should create a b3.0.0 tag so a binary release is created. Any time you want to update which binary release is downloaded by default, you just change ffmpeg-static.binary_release in the package.json.

You can also override it when you install the package with an environment variable ex FFMPEG_BINARY_RELEASE=b4.0.0 npm install

jperl · 2020-01-15T22:22:21Z

I released this as @qawolf/ffmpeg-static and used it in qawolf to check it works. The tests pass and the install works properly.

jperl · 2020-01-20T19:01:02Z

Leaving the url as qawolf so tests pass. Will change it back before merging. I updated it to compress the binaries so the download size is much smaller.

derhuerst

Already looking very good!

Given that we only download one executable on install, having the bin structure and the excludes in package.json#files feel overly complicated. Let's

let install.js just download & index.js just resolve to /ffmpeg//ffmpeg.exe,
let build/index.sh just put the executables in bin (without subdirectories), with the same file names as in the GitHub release.

Thanks for your patience!

.travis.yml

build/index.sh

install.js

test/index.js

jperl · 2020-01-27T16:42:14Z

@derhuerst Thanks for the review 🙏. I made the changes you requested except #33 (comment)

jperl · 2020-01-27T17:05:57Z

@derhuerst made the deltaBytes change. It should be good to go 👍

install.js

jperl · 2020-01-27T17:59:51Z

Good catches. Updated it 👍

jperl · 2020-01-27T18:10:06Z

@derhuerst to wrap this up

create a b* tag to start a release
change the download url from qawolf to eugeneware
update the binary_release in package.json to the tag in step 1

jperl changed the title ~~Release binaries~~ Download platform/architecture specific binaries Jan 14, 2020

jperl force-pushed the release-binaries branch from 77e81b5 to 20b9cd6 Compare January 14, 2020 00:45

derhuerst reviewed Jan 14, 2020

View reviewed changes

.github/workflows/release-binaries.yml Outdated Show resolved Hide resolved

derhuerst reviewed Jan 14, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

jperl force-pushed the release-binaries branch from 82ff5d0 to 5a2a53e Compare January 14, 2020 16:37

derhuerst reviewed Jan 14, 2020

View reviewed changes

jperl added 11 commits January 14, 2020 09:54

add release-binaries workflow

3211f0b

github actions no like pipefail

5471b2a

upload individual files

26e9358

- just try on two for now

upload all binaries

749ab33

create install script that downloads binary

469461c

use package version for download url

7ecd0bb

run release binaries on v tags

25c270f

add npm install script

cdcff17

add progress dependency

707d4a9

remove npm build script

72a0de1

release binaries on b* tag

4d09b24

jperl force-pushed the release-binaries branch from 5a2a53e to 4d09b24 Compare January 14, 2020 16:59

jperl added 3 commits January 14, 2020 19:04

use simple-get in install.js

8738465

allow overriding the binary release with FFMPEG_BINARY_RELEASE env va…

d302fa0

…riable

use official github actions

188e119

jperl force-pushed the release-binaries branch from d611fd4 to 188e119 Compare January 15, 2020 17:57

jperl marked this pull request as ready for review January 15, 2020 18:18

log installation failure if no binary is available

27b9971

jperl force-pushed the release-binaries branch from ae446d4 to 27b9971 Compare January 15, 2020 18:20

change url to eugeneware

a1378e4

release compressed binaries

9425b78

jperl force-pushed the release-binaries branch 4 times, most recently from d1cde19 to 11b43e9 Compare January 20, 2020 18:55

install: download and extract binary

f692aff

jperl force-pushed the release-binaries branch from 11b43e9 to f692aff Compare January 20, 2020 18:57

do not publish tar.gz

f25460e

This was referenced Jan 27, 2020

Clarify possible relationship of binary builds to GPL #8

Closed

split package to reduce install size #20

Closed

derhuerst requested changes Jan 27, 2020

View reviewed changes

.travis.yml Outdated Show resolved Hide resolved

build/index.sh Outdated Show resolved Hide resolved

install.js Outdated Show resolved Hide resolved

install.js Outdated Show resolved Hide resolved

test/index.js Outdated Show resolved Hide resolved

jperl added 5 commits January 27, 2020 09:14

revert auto-format change

dc675e3

reject download with err if there is one

a60bc92

change ffmpeg path to ./...

5402bbc

do not compress, flatten bin folder

c1079b4

update release reference

5f08dbe

do not calculate deltaBytes

167e0d7

derhuerst reviewed Jan 27, 2020

View reviewed changes

install.js Show resolved Hide resolved

derhuerst reviewed Jan 27, 2020

View reviewed changes

install.js Show resolved Hide resolved

process.exit(1) on failures

d80e987

derhuerst approved these changes Jan 27, 2020

View reviewed changes

derhuerst merged commit 503b7ed into eugeneware:master Jan 27, 2020

jperl deleted the release-binaries branch January 27, 2020 18:07

derhuerst mentioned this pull request Feb 11, 2020

support Electron: respect process.env.npm_config_arch & process.env.npm_config_platform #35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Download platform/architecture specific binaries #33

Download platform/architecture specific binaries #33

jperl commented Jan 14, 2020 •

edited

Loading

jperl commented Jan 14, 2020 •

edited

Loading

derhuerst Jan 14, 2020

derhuerst Jan 14, 2020

derhuerst commented Jan 14, 2020

jperl commented Jan 15, 2020 •

edited

Loading

jperl commented Jan 15, 2020 •

edited

Loading

jperl commented Jan 15, 2020 •

edited

Loading

jperl commented Jan 20, 2020

derhuerst left a comment •

edited

Loading

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020 •

edited

Loading

Download platform/architecture specific binaries #33

Download platform/architecture specific binaries #33

Conversation

jperl commented Jan 14, 2020 • edited Loading

jperl commented Jan 14, 2020 • edited Loading

derhuerst Jan 14, 2020

Choose a reason for hiding this comment

derhuerst Jan 14, 2020

Choose a reason for hiding this comment

derhuerst commented Jan 14, 2020

jperl commented Jan 15, 2020 • edited Loading

jperl commented Jan 15, 2020 • edited Loading

jperl commented Jan 15, 2020 • edited Loading

jperl commented Jan 20, 2020

derhuerst left a comment • edited Loading

Choose a reason for hiding this comment

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020

jperl commented Jan 27, 2020 • edited Loading

jperl commented Jan 14, 2020 •

edited

Loading

jperl commented Jan 14, 2020 •

edited

Loading

jperl commented Jan 15, 2020 •

edited

Loading

jperl commented Jan 15, 2020 •

edited

Loading

jperl commented Jan 15, 2020 •

edited

Loading

derhuerst left a comment •

edited

Loading

jperl commented Jan 27, 2020 •

edited

Loading