New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build with codesign takes a long time when 'nodegit' is in my dependencies #1147

Closed
matthewtoast opened this Issue Jan 19, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@matthewtoast

matthewtoast commented Jan 19, 2017

  • Version: 11.3.0
  • Target: Mac

Problem:

I am trying to build a project that has nodegit@0.16.0 as a dependency. When I run:

CSC_LINK={mycertfilepath} CSC_KEY_PASSWORD={mypass} build --mac

The build takes a long time (i.e. never finishes). On my MacBook Pro (Retina, Mid 2012) running MacOS Sierra, the "Rebuilding" and "Packaging" steps take ~5 minutes, which is a bit slow. For the "Signing" step, however, I've waited 30+ minutes and never seen it finish.

With nodegit deleted from "dependencies", the build finishes in seconds.

Details:

A minimal package.json that reproduces the issue looks like this:

{
  "name": "repro-nodegit-electron-build-demo",
  "version": "0.0.0",
  "private": true,
  "description": "Minimal test of building Electron with nodegit and codesigning",
  "main": "index.js",
  "author": "Matthew",
  "engines": {
    "node": "6.6.0",
    "npm": "3.10.3",
    "electron": "1.4.5"
  },
  "build": {
    "appId": "com.electron.nodegitelectronbuilddemo",
    "copyright": "none",
    "productName": "nodegitelectronbuilddemo",
    "forceCodeSigning": true,
    "electronVersion": "1.4.5",
    "mac": {
      "category": "public.app-category.developer-tools"
    }
  },
  "scripts": {
    "build": "build --mac"
  },
  "dependencies": {
    "nodegit": "0.16.0"
  },
  "devDependencies": {
    "electron-builder": "11.3.0"
  }
}

(To repro the signing slowness, you would need to get a developer certificate, etc.)

Hypotheses:

Based on the output of $ ps ax | grep codesign while running the build, my guess is that there are simply so many files to sign that it is just taking a long time. I see it signing a lot of files that seem ancillary.

With that in mind, these are my hypotheses:

  • I am supposed to somehow pre-bundle my code before using electron-builder. (I.e. concatenate my JavaScript files, etc., so that there are fewer files to sign?)

  • I am supposed to configure the "files" build option so that code-signing doesn't have to trawl through a bunch of ancillary files. (It looks like nodegit and its dependencies are rather huge, and there are some test/VCS artifacts also being signed.)

  • I have misconfigured the installation of nodegit somehow, resulting in a big complicated bundle.

  • I am supposed to pre-compile nodegit down to a single bundle before using it here.

  • Waiting over 30 minutes in my scenario is normal. This is simply a fact of life I will have to accept if I want to include nodegit in my project.

  • I am supposed to use the two-package.json setup.

Am I onto something with any of these? Any tips or suggestions?

Apologies if I've missed some important note in the documentation that pertains to this.

@matthewtoast matthewtoast changed the title from Build takes a long time when 'nodegit' is in my dependencies to Build with codesign takes a long time when 'nodegit' is in my dependencies Jan 19, 2017

@matthewtoast

This comment has been minimized.

Show comment
Hide comment
@matthewtoast

matthewtoast Jan 21, 2017

When I ps ax | grep codesign as the codesign step is working, I see output like:

codesign --sign {name_of_my_developer_id_certificate} --force --keychain {target_file_in_temp_folder} node_modules/vendor/libgit2/tests/resources/merge-resolve/.gitted/objects/54/269b3f6ec3d7d4ede24dd350dd5d605495c3ae

So it looks like codesign is going deep into hidden folders, including VCS databases, and signing all of those entries too. That's a lot to sign. Which explains the wait.


My workaround is to add some "files" entries to the build config in my package.json, so that there are fewer files for codesign to crunch:

"build": {
  "files": [
    // ...snip...
    "!node_modules/nodegit",
    "node_modules/nodegit/package.json",
    "node_modules/nodegit/dist/**",
    "node_modules/nodegit/build/Release/nodegit.node",
    // ...snip...
  ]
}

First I exclude the whole nodegit package from my app bundle. Then, I re-include only the parts of nodegit necessary for it to work.

I did this through trial and error, so my file patterns are probably excluding some critical file. But this at least gets the build working, and I don't initially see any problems with require('nodegit') when I launch my packaged app.

matthewtoast commented Jan 21, 2017

When I ps ax | grep codesign as the codesign step is working, I see output like:

codesign --sign {name_of_my_developer_id_certificate} --force --keychain {target_file_in_temp_folder} node_modules/vendor/libgit2/tests/resources/merge-resolve/.gitted/objects/54/269b3f6ec3d7d4ede24dd350dd5d605495c3ae

So it looks like codesign is going deep into hidden folders, including VCS databases, and signing all of those entries too. That's a lot to sign. Which explains the wait.


My workaround is to add some "files" entries to the build config in my package.json, so that there are fewer files for codesign to crunch:

"build": {
  "files": [
    // ...snip...
    "!node_modules/nodegit",
    "node_modules/nodegit/package.json",
    "node_modules/nodegit/dist/**",
    "node_modules/nodegit/build/Release/nodegit.node",
    // ...snip...
  ]
}

First I exclude the whole nodegit package from my app bundle. Then, I re-include only the parts of nodegit necessary for it to work.

I did this through trial and error, so my file patterns are probably excluding some critical file. But this at least gets the build working, and I don't initially see any problems with require('nodegit') when I launch my packaged app.

@matthewtoast

This comment has been minimized.

Show comment
Hide comment
@matthewtoast

matthewtoast Jan 26, 2017

I think it's unlikely there will be a lot of commentary on this one, so I'm going to close it.

matthewtoast commented Jan 26, 2017

I think it's unlikely there will be a lot of commentary on this one, so I'm going to close it.

@develar

This comment has been minimized.

Show comment
Hide comment
@develar

develar Jan 26, 2017

Member

@matthewtoast No, it will be investigated and fixed when I will have time :)

Member

develar commented Jan 26, 2017

@matthewtoast No, it will be investigated and fixed when I will have time :)

@develar

This comment has been minimized.

Show comment
Hide comment
@develar

develar Jan 31, 2017

Member

Solution — do not use npm. Use yarn and execute yarn clean. All junk directories like vendor/libgit2/tests will be removed.

Member

develar commented Jan 31, 2017

Solution — do not use npm. Use yarn and execute yarn clean. All junk directories like vendor/libgit2/tests will be removed.

@develar develar closed this Jan 31, 2017

@develar develar added the question label Jan 31, 2017

@fab1an

This comment has been minimized.

Show comment
Hide comment
@fab1an

fab1an Sep 8, 2017

I have a similar problem, but I don't want to use yarn (I used it for a year, but switch back to npm after various troubles with it).

For me codesign, the final package _CodeSignature contains entries like:

                <key>Resources/app.asar.unpacked/node_modules/lzz-gyp/package.json</key>
		<dict>
			<key>hash</key>
			<data>
			V46JaYPK9FUORjoONzTgww3ovBo=
			</data>
			<key>hash2</key>
			<data>
			c2Zqm6KAgz7BZzpLOD8K9yrP1ocqjCjA0V/jJv58gc0=
			</data>
		</dict>
  • Is this needed?
  • Are signatures for .cpp or .dylib needed?

fab1an commented Sep 8, 2017

I have a similar problem, but I don't want to use yarn (I used it for a year, but switch back to npm after various troubles with it).

For me codesign, the final package _CodeSignature contains entries like:

                <key>Resources/app.asar.unpacked/node_modules/lzz-gyp/package.json</key>
		<dict>
			<key>hash</key>
			<data>
			V46JaYPK9FUORjoONzTgww3ovBo=
			</data>
			<key>hash2</key>
			<data>
			c2Zqm6KAgz7BZzpLOD8K9yrP1ocqjCjA0V/jJv58gc0=
			</data>
		</dict>
  • Is this needed?
  • Are signatures for .cpp or .dylib needed?
@develar

This comment has been minimized.

Show comment
Hide comment
@develar

develar Sep 13, 2017

Member

Yes, such files must be signed. Because it is library.

Member

develar commented Sep 13, 2017

Yes, such files must be signed. Because it is library.

@develar

This comment has been minimized.

Show comment
Hide comment
@develar

develar Sep 13, 2017

Member

You can try to add ignores to files option if some files are not required for you.

Member

develar commented Sep 13, 2017

You can try to add ignores to files option if some files are not required for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment