nodePackages: Enable source patching, patch shebangs, patch versions, fix recursive dependencies runtime #4922
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| inherit src; | ||
|
|
||
|
|
||
| patchPhase = '' |
There was a problem hiding this comment.
What about using postPatch hook instead? Otherwise you lose the ability to easily use patches.
offlinehacker
force-pushed
the
node_shebangs
branch
2 times, most recently
from
ef201fb
to
6239bdc
Compare
|
@lethalman ok, i reused $TMPDIR for making node_modules shims, so i leave unpacked source directory intact |
offlinehacker
force-pushed
the
node_shebangs
branch
from
6239bdc
to
3031c8b
Compare
|
Doesn't using $TMPDIR clutter /tmp too much when used with nix-shell? Have to try. |
|
@lethalman well, hopefully not, i'm using $TMPDIR, for it's purpose. |
|
ShellHook should work as it is |
|
Also there's one more thing i'm unsure. If a source is a directory(if fetchgit is used), then sometimes you need run nodejs's post publish hooks(which get only run if source is a folder) and not repack the archive. |
|
Note in my original PR I also handled the case when src is a directory. In fact ghost doesn't build: No idea how the src can be a directory instead of a tgz :S |
|
Ok, now i handled a case where source is directory, how is that the gost is directory with tgz extension? |
|
Ah i can see the problem, patchSoruce is used. We should throw that out npm2nix, that is wrong way of doing things. |
|
Look into node-packages-generated.nix |
|
Ok, ghost builds now fine |
|
@offlinehacker ah then the shebangs patch could simply go in patchSource. |
|
@offlinehacker seems to me that what we are doing is wrong. This stuff should just go in patchSource. |
|
why? |
|
@offlinehacker because it's exactly what patchSource was made for :) just lacking the shebangs patch. |
|
Well i do not agree, patchSource is only called in speciffic occasions |
|
Patching should be avalible anytime user wants to patch source |
|
And it should be in patchPhase not some additonal magic function that tends to break. |
|
This only covers one use-case, and solves only one problem, this implementation covers multiple problems and inside nix expression for general use cases. |
|
@offlinehacker perhaps we can move that patchSource in postPatch and make patchSource a no-op until npm2nix is updated? That's another issue though. |
|
@lethalman already done in my last commit |
|
@offlinehacker anyway $TMPDIR is /tmp when building in nix-shell. Can we instead please create a couple of directories under pwd? |
|
@lethalman i will probably just use mktemp -d |
|
@offlinehacker ./ in nix is already a temp dir in nix-build... I don't get what's wrong in creating a couple of directories under ./ |
|
@offlinehacker also not sure, but does this PR contemplate multiple src? |
|
@lethalman i haven't seen multiple srcs? Is there any usage for multiple srcs, any example? |
|
I think having multiple srcs is wrong approach, but is there any real use case npm uses where we need multiple srcs? |
|
No idea I've seen it's an array. If there's no multiple srcs it's ok, but On Mon, Nov 10, 2014 at 10:38 PM, Jaka Hudoklin notifications@github.com
www.debian.org - The Universal Operating System |
|
Ok first, let's find out what was the case with multiple sources, i can see in npm2nix code that there is such scenario, but what is it used for. I guess in that case both sources are extracted, but i would guess they are installed separatelly? |
offlinehacker
force-pushed
the
node_shebangs
branch
2 times, most recently
from
eb62365
to
1bad9ff
Compare
offlinehacker
changed the title
- reuse unpack and patch phase - patch shebangs of source code - properly patch depdency versions (thanks @svanderburg) - add patching for github version names - add some inline docs - fix recursive depdencies runtime handling by copying files instead of symlinking if package has recursive dependencies
|
I've also fixed recursive dependencies, i'm merging this pull request, because it fixes a lot of broken packages. I've tried and all except karma and webdrvr don't build, becuase webdrvr can't download selenium, and karma, because currently we can't handle optional dependencies(fsevents). |
offlinehacker
force-pushed
the
node_shebangs
branch
from
1bad9ff
to
86744fe
Compare
offlinehacker
added a commit
that referenced
this pull request
Dec 26, 2014
nodePackages: Enable source patching, patch shebangs, patch versions, fix recursive dependencies runtime
mweinelt
added a commit
to mweinelt/nixpkgs
that referenced
this pull request
Mar 8, 2020
Version 1.1.11 (2020-03-08)
Compatibility notes:
When upgrading from borg 1.0.x to 1.1.x, please note:
read all the compatibility notes for 1.1.0*, starting from 1.1.0b1.
borg upgrade: you do not need to and you also should not run it.
borg might ask some security-related questions once after upgrading. You can answer them either manually or via environment variable. One known case is if you use unencrypted repositories, then it will ask about a unknown unencrypted repository one time.
your first backup with 1.1.x might be significantly slower (it might completely read, chunk, hash a lot files) - this is due to the --files-cache mode change (and happens every time you change mode). You can avoid the one-time slowdown by using the pre-1.1.0rc4-compatible mode (but that is less safe for detecting changed files than the default). See the --files-cache docs for details.
1.1.11 removes WSL autodetection (Windows 10 Subsystem for Linux). If WSL still has a problem with sync_file_range, you need to set BORG_WORKAROUNDS=basesyncfile in the borg process environment to work around the WSL issue.
Fixes:
fixed potential index corruption / data loss issue due to bug in hashindex_set, NixOS#4829 Please read and follow the more detailled notes close to the top of this document.
upgrade bundled xxhash to 0.7.3, NixOS#4891 0.7.2 is the minimum requirement for correct operations on ARMv6 in non-fixup mode, where unaligned memory accesses cause bus errors. 0.7.3 adds some speedups and libxxhash 0.7.3 even has a pkg-config file now.
upgrade bundled lz4 to 1.9.2
upgrade bundled zstd to 1.4.4
fix crash when upgrading erroneous hints file, NixOS#4922
extract:
fix KeyError for "partial" extraction, NixOS#4607
fix "partial" extract for hardlinked contentless file types, NixOS#4725
fix preloading for old (0.xx) remote servers, NixOS#4652
fix confusing output of borg extract --list --strip-components, NixOS#4934
delete: after double-force delete, warn about necessary repair, NixOS#4704
create: give invalid repo error msg if repo config not found, NixOS#4411
mount: fix FUSE mount missing st_birthtime, NixOS#4763 NixOS#4767
check: do not stumble over invalid item key, NixOS#4845
info: if the archive doesn't exist, print a pretty message, NixOS#4793
SecurityManager.known(): check all files, NixOS#4614
Repository.open: use stat() to check for repo dir, NixOS#4695
Repository.check_can_create_repository: use stat() to check, NixOS#4695
fix invalid archive error message
fix optional/non-optional location arg, NixOS#4541
commit-time free space calc: ignore bad compact map entries, NixOS#4796
ignore EACCES (errno 13) when hardlinking the old config, NixOS#4730
--prefix / -P: fix processing, avoid argparse issue, NixOS#4769
New features:
enable placeholder usage in all extra archive arguments
new BORG_WORKAROUNDS mechanism, basesyncfile, NixOS#4710
recreate: support --timestamp option, NixOS#4745
support platforms without os.link (e.g. Android with Termux), NixOS#4901 if we don't have os.link, we just extract another copy instead of making a hardlink.
support linux platforms without sync_file_range (e.g. Android 7 with Termux), NixOS#4905
Other:
ignore --stats when given with --dry-run, but continue, NixOS#4373
add some ProgressIndicator msgids to code / fix docs, NixOS#4935
elaborate on "Calculating size" message
argparser: always use REPOSITORY in metavar, also use more consistent help phrasing.
check: improve error output for matching index size, see NixOS#4829
docs:
changelog: add advisory about hashindex_set bug NixOS#4829
better describe BORG_SECURITY_DIR, BORG_CACHE_DIR, NixOS#4919
infos about cache security assumptions, NixOS#4900
add FAQ describing difference between a local repo vs. repo on a server.
document how to test exclusion patterns without performing an actual backup
timestamps in the files cache are now usually ctime, NixOS#4583
fix bad reference to borg compact (does not exist in 1.1), NixOS#4660
create: borg 1.1 is not future any more
extract: document limitation "needs empty destination", NixOS#4598
how to supply a passphrase, use crypto devices, NixOS#4549
fix osxfuse github link in installation docs
add example of exclude-norecurse rule in help patterns
update macOS Brew link
add note about software for automating backups, NixOS#4581
AUTHORS: mention copyright+license for bundled msgpack
fix various code blocks in the docs, NixOS#4708
updated docs to cover use of temp directory on remote, NixOS#4545
add restore docs, NixOS#4670
add a pull backup / push restore how-to, NixOS#1552
add FAQ how to retain original paths, NixOS#4532
explain difference between --exclude and --pattern, NixOS#4118
add FAQs for SSH connection issues, NixOS#3866
improve password FAQ, NixOS#4591
reiterate that 'file cache names are absolute' in FAQ
tests:
cope with ANY error when importing pytest into borg.testsuite, NixOS#4652
fix broken test that relied on improper zlib assumptions
test_fuse: filter out selinux xattrs, NixOS#4574
travis / vagrant:
misc python versions removed / changed (due to openssl 1.1 compatibility) or added (3.7 and 3.8, for better borg compatibility testing)
binary building is on python 3.5.9 now
vagrant:
add new boxes: ubuntu 18.04 and 20.04, debian 10
update boxes: openindiana, darwin, netbsd
remove old boxes: centos 6
darwin: updated osxfuse to 3.10.4
use debian/ubuntu pip/virtualenv packages
rather use python 3.6.2 than 3.6.0, fixes coverage/sqlite3 issue
use requirements.d/development.lock.txt to avoid compat issues
travis:
darwin: backport some install code / order from master
remove deprecated keyword "sudo" from travis config
allow osx builds to fail, NixOS#4955 this is due to travis-ci frequently being so slow that the OS X builds just fail because they exceed 50 minutes and get killed by travis.
mweinelt
added a commit
to mweinelt/nixpkgs
that referenced
this pull request
Mar 8, 2020
Version 1.1.11 (2020-03-08)
Compatibility notes:
When upgrading from borg 1.0.x to 1.1.x, please note:
read all the compatibility notes for 1.1.0*, starting from 1.1.0b1.
borg upgrade: you do not need to and you also should not run it.
borg might ask some security-related questions once after upgrading. You can answer them either manually or via environment variable. One known case is if you use unencrypted repositories, then it will ask about a unknown unencrypted repository one time.
your first backup with 1.1.x might be significantly slower (it might completely read, chunk, hash a lot files) - this is due to the --files-cache mode change (and happens every time you change mode). You can avoid the one-time slowdown by using the pre-1.1.0rc4-compatible mode (but that is less safe for detecting changed files than the default). See the --files-cache docs for details.
1.1.11 removes WSL autodetection (Windows 10 Subsystem for Linux). If WSL still has a problem with sync_file_range, you need to set BORG_WORKAROUNDS=basesyncfile in the borg process environment to work around the WSL issue.
Fixes:
fixed potential index corruption / data loss issue due to bug in hashindex_set, NixOS#4829 Please read and follow the more detailled notes close to the top of this document.
upgrade bundled xxhash to 0.7.3, NixOS#4891 0.7.2 is the minimum requirement for correct operations on ARMv6 in non-fixup mode, where unaligned memory accesses cause bus errors. 0.7.3 adds some speedups and libxxhash 0.7.3 even has a pkg-config file now.
upgrade bundled lz4 to 1.9.2
upgrade bundled zstd to 1.4.4
fix crash when upgrading erroneous hints file, NixOS#4922
extract:
fix KeyError for "partial" extraction, NixOS#4607
fix "partial" extract for hardlinked contentless file types, NixOS#4725
fix preloading for old (0.xx) remote servers, NixOS#4652
fix confusing output of borg extract --list --strip-components, NixOS#4934
delete: after double-force delete, warn about necessary repair, NixOS#4704
create: give invalid repo error msg if repo config not found, NixOS#4411
mount: fix FUSE mount missing st_birthtime, NixOS#4763 NixOS#4767
check: do not stumble over invalid item key, NixOS#4845
info: if the archive doesn't exist, print a pretty message, NixOS#4793
SecurityManager.known(): check all files, NixOS#4614
Repository.open: use stat() to check for repo dir, NixOS#4695
Repository.check_can_create_repository: use stat() to check, NixOS#4695
fix invalid archive error message
fix optional/non-optional location arg, NixOS#4541
commit-time free space calc: ignore bad compact map entries, NixOS#4796
ignore EACCES (errno 13) when hardlinking the old config, NixOS#4730
--prefix / -P: fix processing, avoid argparse issue, NixOS#4769
New features:
enable placeholder usage in all extra archive arguments
new BORG_WORKAROUNDS mechanism, basesyncfile, NixOS#4710
recreate: support --timestamp option, NixOS#4745
support platforms without os.link (e.g. Android with Termux), NixOS#4901 if we don't have os.link, we just extract another copy instead of making a hardlink.
support linux platforms without sync_file_range (e.g. Android 7 with Termux), NixOS#4905
Other:
ignore --stats when given with --dry-run, but continue, NixOS#4373
add some ProgressIndicator msgids to code / fix docs, NixOS#4935
elaborate on "Calculating size" message
argparser: always use REPOSITORY in metavar, also use more consistent help phrasing.
check: improve error output for matching index size, see NixOS#4829
docs:
changelog: add advisory about hashindex_set bug NixOS#4829
better describe BORG_SECURITY_DIR, BORG_CACHE_DIR, NixOS#4919
infos about cache security assumptions, NixOS#4900
add FAQ describing difference between a local repo vs. repo on a server.
document how to test exclusion patterns without performing an actual backup
timestamps in the files cache are now usually ctime, NixOS#4583
fix bad reference to borg compact (does not exist in 1.1), NixOS#4660
create: borg 1.1 is not future any more
extract: document limitation "needs empty destination", NixOS#4598
how to supply a passphrase, use crypto devices, NixOS#4549
fix osxfuse github link in installation docs
add example of exclude-norecurse rule in help patterns
update macOS Brew link
add note about software for automating backups, NixOS#4581
AUTHORS: mention copyright+license for bundled msgpack
fix various code blocks in the docs, NixOS#4708
updated docs to cover use of temp directory on remote, NixOS#4545
add restore docs, NixOS#4670
add a pull backup / push restore how-to, NixOS#1552
add FAQ how to retain original paths, NixOS#4532
explain difference between --exclude and --pattern, NixOS#4118
add FAQs for SSH connection issues, NixOS#3866
improve password FAQ, NixOS#4591
reiterate that 'file cache names are absolute' in FAQ
tests:
cope with ANY error when importing pytest into borg.testsuite, NixOS#4652
fix broken test that relied on improper zlib assumptions
test_fuse: filter out selinux xattrs, NixOS#4574
travis / vagrant:
misc python versions removed / changed (due to openssl 1.1 compatibility) or added (3.7 and 3.8, for better borg compatibility testing)
binary building is on python 3.5.9 now
vagrant:
add new boxes: ubuntu 18.04 and 20.04, debian 10
update boxes: openindiana, darwin, netbsd
remove old boxes: centos 6
darwin: updated osxfuse to 3.10.4
use debian/ubuntu pip/virtualenv packages
rather use python 3.6.2 than 3.6.0, fixes coverage/sqlite3 issue
use requirements.d/development.lock.txt to avoid compat issues
travis:
darwin: backport some install code / order from master
remove deprecated keyword "sudo" from travis config
allow osx builds to fail, NixOS#4955 this is due to travis-ci frequently being so slow that the OS X builds just fail because they exceed 50 minutes and get killed by travis.
(cherry picked from commit dbff9b5)
mweinelt
added a commit
to mweinelt/nixpkgs
that referenced
this pull request
Mar 8, 2020
Version 1.1.11 (2020-03-08)
Compatibility notes:
When upgrading from borg 1.0.x to 1.1.x, please note:
read all the compatibility notes for 1.1.0*, starting from 1.1.0b1.
borg upgrade: you do not need to and you also should not run it.
borg might ask some security-related questions once after upgrading. You can answer them either manually or via environment variable. One known case is if you use unencrypted repositories, then it will ask about a unknown unencrypted repository one time.
your first backup with 1.1.x might be significantly slower (it might completely read, chunk, hash a lot files) - this is due to the --files-cache mode change (and happens every time you change mode). You can avoid the one-time slowdown by using the pre-1.1.0rc4-compatible mode (but that is less safe for detecting changed files than the default). See the --files-cache docs for details.
1.1.11 removes WSL autodetection (Windows 10 Subsystem for Linux). If WSL still has a problem with sync_file_range, you need to set BORG_WORKAROUNDS=basesyncfile in the borg process environment to work around the WSL issue.
Fixes:
fixed potential index corruption / data loss issue due to bug in hashindex_set, NixOS#4829 Please read and follow the more detailled notes close to the top of this document.
upgrade bundled xxhash to 0.7.3, NixOS#4891 0.7.2 is the minimum requirement for correct operations on ARMv6 in non-fixup mode, where unaligned memory accesses cause bus errors. 0.7.3 adds some speedups and libxxhash 0.7.3 even has a pkg-config file now.
upgrade bundled lz4 to 1.9.2
upgrade bundled zstd to 1.4.4
fix crash when upgrading erroneous hints file, NixOS#4922
extract:
fix KeyError for "partial" extraction, NixOS#4607
fix "partial" extract for hardlinked contentless file types, NixOS#4725
fix preloading for old (0.xx) remote servers, NixOS#4652
fix confusing output of borg extract --list --strip-components, NixOS#4934
delete: after double-force delete, warn about necessary repair, NixOS#4704
create: give invalid repo error msg if repo config not found, NixOS#4411
mount: fix FUSE mount missing st_birthtime, NixOS#4763 NixOS#4767
check: do not stumble over invalid item key, NixOS#4845
info: if the archive doesn't exist, print a pretty message, NixOS#4793
SecurityManager.known(): check all files, NixOS#4614
Repository.open: use stat() to check for repo dir, NixOS#4695
Repository.check_can_create_repository: use stat() to check, NixOS#4695
fix invalid archive error message
fix optional/non-optional location arg, NixOS#4541
commit-time free space calc: ignore bad compact map entries, NixOS#4796
ignore EACCES (errno 13) when hardlinking the old config, NixOS#4730
--prefix / -P: fix processing, avoid argparse issue, NixOS#4769
New features:
enable placeholder usage in all extra archive arguments
new BORG_WORKAROUNDS mechanism, basesyncfile, NixOS#4710
recreate: support --timestamp option, NixOS#4745
support platforms without os.link (e.g. Android with Termux), NixOS#4901 if we don't have os.link, we just extract another copy instead of making a hardlink.
support linux platforms without sync_file_range (e.g. Android 7 with Termux), NixOS#4905
Other:
ignore --stats when given with --dry-run, but continue, NixOS#4373
add some ProgressIndicator msgids to code / fix docs, NixOS#4935
elaborate on "Calculating size" message
argparser: always use REPOSITORY in metavar, also use more consistent help phrasing.
check: improve error output for matching index size, see NixOS#4829
docs:
changelog: add advisory about hashindex_set bug NixOS#4829
better describe BORG_SECURITY_DIR, BORG_CACHE_DIR, NixOS#4919
infos about cache security assumptions, NixOS#4900
add FAQ describing difference between a local repo vs. repo on a server.
document how to test exclusion patterns without performing an actual backup
timestamps in the files cache are now usually ctime, NixOS#4583
fix bad reference to borg compact (does not exist in 1.1), NixOS#4660
create: borg 1.1 is not future any more
extract: document limitation "needs empty destination", NixOS#4598
how to supply a passphrase, use crypto devices, NixOS#4549
fix osxfuse github link in installation docs
add example of exclude-norecurse rule in help patterns
update macOS Brew link
add note about software for automating backups, NixOS#4581
AUTHORS: mention copyright+license for bundled msgpack
fix various code blocks in the docs, NixOS#4708
updated docs to cover use of temp directory on remote, NixOS#4545
add restore docs, NixOS#4670
add a pull backup / push restore how-to, NixOS#1552
add FAQ how to retain original paths, NixOS#4532
explain difference between --exclude and --pattern, NixOS#4118
add FAQs for SSH connection issues, NixOS#3866
improve password FAQ, NixOS#4591
reiterate that 'file cache names are absolute' in FAQ
tests:
cope with ANY error when importing pytest into borg.testsuite, NixOS#4652
fix broken test that relied on improper zlib assumptions
test_fuse: filter out selinux xattrs, NixOS#4574
travis / vagrant:
misc python versions removed / changed (due to openssl 1.1 compatibility) or added (3.7 and 3.8, for better borg compatibility testing)
binary building is on python 3.5.9 now
vagrant:
add new boxes: ubuntu 18.04 and 20.04, debian 10
update boxes: openindiana, darwin, netbsd
remove old boxes: centos 6
darwin: updated osxfuse to 3.10.4
use debian/ubuntu pip/virtualenv packages
rather use python 3.6.2 than 3.6.0, fixes coverage/sqlite3 issue
use requirements.d/development.lock.txt to avoid compat issues
travis:
darwin: backport some install code / order from master
remove deprecated keyword "sudo" from travis config
allow osx builds to fail, NixOS#4955 this is due to travis-ci frequently being so slow that the OS X builds just fail because they exceed 50 minutes and get killed by travis.
(cherry picked from commit dbff9b5)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This enables patching of node packages and adds patch for shebangs and patch for versions. This pull request replaces https://github.com/NixOS/nixpkgs/pull/4850/files