fix: github private repo differential updates #6198
Conversation
🦋 Changeset detectedLatest commit: 34624b4 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| private getVersionUrl(version: string): URL { | ||
| const tagName = this.options.vPrefixedTagName === false ? version : `v${version}` | ||
| return new URL(`/repos/${this.options.owner}/${this.options.repo}/releases/tags/${tagName}`, this.baseApiUrl) | ||
| } |
There was a problem hiding this comment.
Can you test using this?
return newUrlFromBase(`/repos/${this.options.owner}/${this.options.repo}/releases/tags/${tagName}`, this.baseApiUrl)
There was a problem hiding this comment.
This works as well! The only issue I'm still running into is the following:
[2021-08-30 07:52:19.230] [info] Checking for update
[2021-08-30 07:52:20.194] [info] Found version 0.0.2 (url: Electron-Update-Test-Setup-0.0.2.exe)
[2021-08-30 07:52:20.194] [info] Downloading update from Electron-Update-Test-Setup-0.0.2.exe
[2021-08-30 07:52:20.197] [info] Download block maps (old: "https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43620195", new: https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43620268)
[2021-08-30 07:52:20.371] [info] File has 35 changed blocks
[2021-08-30 07:52:20.381] [info] Full: 57,990.1 KB, To download: 764.79 KB (1%)
[2021-08-30 07:52:20.384] [info] Differential download: https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43620267
[2021-08-30 07:52:21.401] [error] Cannot download differentially, fallback to full download: Error: sha512 checksum mismatch, expected JhtI6yn2plziVTLgNT6nN7OS1eA6GQlL3zsKsymDwkv/EjtOQqV0kpJP6RXYdabYlvJUxsFMB0oCKQWV4HsGyQ==, got OpLHghzGPj+9sbeUN/wxkIazBxwtbVuxMZKbNz2YRMRxAICu9jhj2cgfLVnaLYajgLAI7kHZJERRYATnQfaCZQ==
at Object.newError (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\builder-util-runtime\out\index.js:47:19)
at DigestTransform.validate (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\builder-util-runtime\out\httpExecutor.js:355:27)
at WriteStream.<anonymous> (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\electron-updater\out\differentialDownloader\DifferentialDownloader.js:133:41)
at WriteStream.emit (events.js:315:20)
at emitCloseNT (internal/streams/destroy.js:87:10)
at processTicksAndRejections (internal/process/task_queues.js:79:21)
[2021-08-30 07:52:26.008] [info] New version 0.0.2 has been downloaded to C:\Users\NGrey\AppData\Local\electron update test-updater\pending\update.exe
There was a problem hiding this comment.
Can you try that again and see what files it is comparing?
I'm curious if it's comparing a blockmap checksum with the executable checksum on accident.
You can add logging to the autoupdater via:
const log = require("electron-log")
log.transports.file.level = "debug"
autoUpdater.logger = log
There was a problem hiding this comment.
Here is the output for debug. I think it's important to note that a Public github repo doesn't have any checksum error. It's almost as if the info we're trying to get from the github API isn't the same. I've pasted in both debugs for both private and public.
PRIVATE GITHUB REPO
[2021-08-31 07:47:19.819] [info] Checking for update
[2021-08-31 07:47:19.838] [info] Generated new staging user ID: 8e2c8251-81a3-5053-9660-b4c8a52d305e
[2021-08-31 07:47:20.881] [info] Found version 0.0.2 (url: ElectronUpdateTestSetup.exe)
[2021-08-31 07:47:20.881] [info] Downloading update from ElectronUpdateTestSetup.exe
[2021-08-31 07:47:20.883] [debug] updater cache dir: C:\Users\NGrey\AppData\Local\electron update test-updater
[2021-08-31 07:47:20.885] [info] Download block maps (old: "https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43663242", new: https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43663314)
[2021-08-31 07:47:21.096] [info] File has 35 changed blocks
[2021-08-31 07:47:21.097] [debug] [
{
"kind": 0,
"start": 0,
"end": 211730
},
{
"kind": 1,
"start": 211730,
"end": 267840
},
{
"kind": 0,
"start": 267835,
"end": 314321
},
{
"kind": 1,
"start": 314326,
"end": 336894
},
{
"kind": 0,
"start": 336889,
"end": 10484878
},
{
"kind": 1,
"start": 10484883,
"end": 10951925
},
{
"kind": 0,
"start": 10952018,
"end": 49629754
},
{
"kind": 1,
"start": 49629661,
"end": 49645896
},
{
"kind": 0,
"start": 49645989,
"end": 49672900
},
{
"kind": 1,
"start": 49672807,
"end": 49782511
},
{
"kind": 0,
"start": 49782625,
"end": 58935645
},
{
"kind": 1,
"start": 58935531,
"end": 58968299
},
{
"kind": 0,
"start": 58968413,
"end": 59201214
},
{
"kind": 1,
"start": 59201100,
"end": 59210764
},
{
"kind": 0,
"start": 59210878,
"end": 59270290
},
{
"kind": 1,
"start": 59270176,
"end": 59381894
}
]
[2021-08-31 07:47:21.106] [info] Full: 57,990.13 KB, To download: 806.45 KB (1%)
[2021-08-31 07:47:21.109] [info] Differential download: https://api.github.com/repos/NGrey5/electron-update-test/releases/assets/43663315
[2021-08-31 07:47:21.112] [debug] download range: bytes=211730-267839
[2021-08-31 07:47:21.213] [debug] download range: bytes=314326-336893
[2021-08-31 07:47:21.342] [debug] download range: bytes=10484883-10951924
[2021-08-31 07:47:21.578] [debug] download range: bytes=49629661-49645895
[2021-08-31 07:47:21.666] [debug] download range: bytes=49672807-49782510
[2021-08-31 07:47:21.847] [debug] download range: bytes=58935531-58968298
[2021-08-31 07:47:21.956] [debug] download range: bytes=59201100-59210763
[2021-08-31 07:47:22.054] [debug] download range: bytes=59270176-59381893
[2021-08-31 07:47:22.145] [error] Cannot download differentially, fallback to full download: Error: sha512 checksum mismatch, expected JhtI6yn2plziVTLgNT6nN7OS1eA6GQlL3zsKsymDwkv/EjtOQqV0kpJP6RXYdabYlvJUxsFMB0oCKQWV4HsGyQ==, got OpLHghzGPj+9sbeUN/wxkIazBxwtbVuxMZKbNz2YRMRxAICu9jhj2cgfLVnaLYajgLAI7kHZJERRYATnQfaCZQ==
at Object.newError (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\builder-util-runtime\out\index.js:47:19)
at DigestTransform.validate (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\builder-util-runtime\out\httpExecutor.js:355:27)
at WriteStream.<anonymous> (C:\Users\NGrey\AppData\Local\Programs\Electron Update Test\resources\app.asar\node_modules\electron-updater\out\differentialDownloader\DifferentialDownloader.js:133:41)
at WriteStream.emit (events.js:315:20)
at emitCloseNT (internal/streams/destroy.js:87:10)
at processTicksAndRejections (internal/process/task_queues.js:79:21)
[2021-08-31 07:47:27.166] [info] New version 0.0.2 has been downloaded to C:\Users\NGrey\AppData\Local\electron update test-updater\pending\update.exe
PUBLIC GITHUB REPO
[2021-08-31 07:57:41.365] [info] Checking for update
[2021-08-31 07:57:42.525] [info] Found version 0.0.2 (url: ElectronUpdateTestSetup.exe)
[2021-08-31 07:57:42.526] [info] Downloading update from ElectronUpdateTestSetup.exe
[2021-08-31 07:57:42.527] [debug] updater cache dir: C:\Users\NGrey\AppData\Local\electron update test-updater
[2021-08-31 07:57:42.529] [info] Download block maps (old: "https://github.com/ngrey5/electron-update-test/releases/download/v0.0.1/ElectronUpdateTestSetup.exe.blockmap", new: https://github.com/ngrey5/electron-update-test/releases/download/v0.0.2/ElectronUpdateTestSetup.exe.blockmap)
[2021-08-31 07:57:42.764] [info] File has 35 changed blocks
[2021-08-31 07:57:42.764] [debug] [
{
"kind": 0,
"start": 0,
"end": 211730
},
{
"kind": 1,
"start": 211730,
"end": 267833
},
{
"kind": 0,
"start": 267833,
"end": 314319
},
{
"kind": 1,
"start": 314319,
"end": 336887
},
{
"kind": 0,
"start": 336887,
"end": 10484876
},
{
"kind": 1,
"start": 10484876,
"end": 10909326
},
{
"kind": 0,
"start": 10909410,
"end": 49629776
},
{
"kind": 1,
"start": 49629692,
"end": 49645927
},
{
"kind": 0,
"start": 49646011,
"end": 49672922
},
{
"kind": 1,
"start": 49672838,
"end": 49782542
},
{
"kind": 0,
"start": 49782647,
"end": 58935667
},
{
"kind": 1,
"start": 58935562,
"end": 58976828
},
{
"kind": 0,
"start": 58976932,
"end": 59201235
},
{
"kind": 1,
"start": 59201131,
"end": 59210795
},
{
"kind": 0,
"start": 59210899,
"end": 59270311
},
{
"kind": 1,
"start": 59270207,
"end": 59381925
}
]
[2021-08-31 07:57:42.774] [info] Full: 57,990.16 KB, To download: 773.15 KB (1%)
[2021-08-31 07:57:42.777] [info] Differential download: https://github.com/ngrey5/electron-update-test/releases/download/v0.0.2/ElectronUpdateTestSetup.exe
[2021-08-31 07:57:42.780] [debug] download range: bytes=211730-267832
[2021-08-31 07:57:43.073] [info] Redirect to https://github-releases.githubusercontent.com/401702721/fa673fdd-8c19-46fb-a420-98ed779a59b0
[2021-08-31 07:57:43.137] [debug] download range: bytes=314319-336886
[2021-08-31 07:57:43.184] [debug] download range: bytes=10484876-10909325
[2021-08-31 07:57:43.396] [debug] download range: bytes=49629692-49645926
[2021-08-31 07:57:43.616] [debug] download range: bytes=49672838-49782541
[2021-08-31 07:57:43.663] [debug] download range: bytes=58935562-58976827
[2021-08-31 07:57:43.714] [debug] download range: bytes=59201131-59210794
[2021-08-31 07:57:43.721] [debug] download range: bytes=59270207-59381924
[2021-08-31 07:57:43.734] [info] New version 0.0.2 has been downloaded to C:\Users\NGrey\AppData\Local\electron update test-updater\pending\ElectronUpdateTestSetup.exe
There was a problem hiding this comment.
@NGrey5 I'm not sure I understand what this PR is solving. Is the current draft fixing private github downloads or did we discover a new issue (i.e. the new issue is checksum mismatch)
There was a problem hiding this comment.
@mmaietta The PR is solving differential downloads with private github repos. Before it wouldn't download the proper files (blockmap.) With the changes currently made, it now downloads and compares the correct files. The checksum mismatch is an issue we now have after those changes have been made. It only happens with a private repo, so I would say it technically should be grouped together; but I could be wrong.
|
If anyone wants to try and help figure out the checksum issue; I would appreciate it! I haven't been able to figure out anything that solves it. |
|
I have developed a better delta update mechanism based on binary diffing for the NSIS target. It's being deployed in production with thousands of users and running smoothly. The delta update mechanism based on the blockmap approach uses a very good amount of bandwidth. The binary diffing approach works better and downloads only the diff, generally less than 1MB to update the app. |
|
@sandeep1995 would you be willing to contribute that logic to electron-builder? Or share the logic via gist or sample electron-quick-start repo? |
|
I don't understand why the private repo SHA512 can't match. Maybe there is a problem with the latest file and blockmap file in the process of building |
|
@NGrey5 Were you able to figure out the solution for private GitHub repos? |
|
@Supriy-codeBreaker I haven't made any further progress on this due to the checksum mismatch error. I'm awaiting someone with a little more knowledge of this to possibly help out. |
|
As I mentioned earlier here #6198 (comment) If you guys are looking for a true delta builder and updater, please checkout |
|
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward? This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
Following the issues discussed in #6184