Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Python 3 #1337

Closed
cclauss opened this issue Nov 14, 2017 · 69 comments

Comments

@cclauss
Copy link
Contributor

commented Nov 14, 2017

See #1150, #1335, #1336, and https://travis-ci.com/nodejs/node/builds/79706150 (nodejs/node#21942)

  • Node Version:
  • Platform:
  • Compiler:
  • Module:
Verbose output (from npm or node-gyp):

@richardlau

This comment has been minimized.

Copy link
Member

commented Nov 14, 2017

Historically the blocker to this has been the gyp dependency and normally changes should be upstreamed there unless we're planning on forking gyp in which case we would want our own gyp project so that we can share the changes with the main Node.js project.

cc @nodejs/gyp

@mthacker801

This comment has been minimized.

Copy link

commented Feb 8, 2018

I cannot successfully install Angular on my mac. I keep getting gyp error and when I look at gyp-mac-tool it says:
gyp verb command configure []
gyp verb check python checking for Python executable "python2" in the PATH
gyp verb which failed Error: not found: python2
gyp verb which failed at getNotFoundError (/Users/thackerfam/AngularProj/angular-cli/node_modules/which/which.js:13:12)

I have had python 3 installed forever and do not want python 2 as it can cause havoc to have both installed as they are not compatible with one another. If this is the reason for my inability to use Angular, what should I do?

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2018

On your Mac, what happens when you do:

  • python2 --version
  • /usr/bin/python --version
  • /usr/bin/python2 --version
  • /usr/bin/python2.7 --version

I would highly recommend Homebrew as essential if you are doing real development on a Mac.

Brew allows you to easily install both Python 2 and Python3 with: brew install python python@2

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Feb 8, 2018

A few Python 2 issues on gyp:

$ python2 -m flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./gyptest.py:61:47: F821 undefined name 'opts'
    extra_path = [os.path.abspath(p) for p in opts.path]  # <-- 'opts' should be changed to 'args'
                                              ^
./pylib/gyp/generator/make.py:1637:50: F821 undefined name 'target'
      print "WARNING: no output for", self.type, target  # <-- 'target' should be changes to 'self.target'
                                                 ^
@SatishMHiremath

This comment has been minimized.

Copy link

commented Feb 17, 2018

I have faced and fixed this issue relates node-sass and python error.Try installing latest angular cli command with the steps below

Set the system variable path as well:
C:\Users\windowsusername\AppData\Roaming\npm

C:\Users\windowsusername\AppData\Roaming\npm\node_modules@angular\cli\bin;

C:\Program Files\nodejs;

Then start using angular cli command prompt using:
angular client project folder path>npm install -g @latest

angular client project folder path>npm install -g node-gyp

@stanleyxu2005

This comment has been minimized.

Copy link

commented Feb 26, 2018

It's a bit embraced that the original gyp project relies on python 2, and after couple of years, the patch is getting too big so that no people is dare to review and merge it. Need to give the gyp community more feedback to move this forward.

@AdamMajer

This comment has been minimized.

Copy link

commented Mar 2, 2018

Is there any interest in this? Python 2 is starting to get removed from distributions at this point.

@timoxley

This comment has been minimized.

Copy link

commented Mar 3, 2018

Python 2 is starting to get removed from distributions at this point.

Correct e.g. brew is now installing/upgrading to Python 3 by default.

The following just worked for me as I already had a python2.7 executable installed via brew:

npm config set python python2.7

However, if you don't have a python2.7 executable, I suspect you can do something like:

brew install python@2
# follow the instructions printed after installing `python@2` to get python 2 on your PATH 
# e.g. echo 'export PATH="/usr/local/opt/python@2/bin:$PATH"' >> ~/.bashrc
npm config set python python2.7
@NetOperatorWibby

This comment has been minimized.

Copy link

commented Mar 24, 2018

Tim's method should go in the README. I use ZSH and there was a message mentioned Xcode when I installed Python 2 so my process was as follows:

xcode-select --install

brew install python@2
echo 'export PATH="/usr/local/opt/python@2/bin:$PATH"' >> ~/.zshrc
npm config set python python2.7
@ioquatix

This comment has been minimized.

Copy link

commented May 4, 2018

For some reason, npm config set python python2.7 stopped working for me. Even after setting it, I get the error that node-gyp is trying to run python 3.6

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented May 4, 2018

What does python2.7 --version return on your machine?

@ioquatix

This comment has been minimized.

Copy link

commented May 4, 2018

> python2.7 --version
Python 2.7.14
@ioquatix

This comment has been minimized.

Copy link

commented May 4, 2018

Here is the full error

^_^ > apm install      
Installing modules ✗
> node-pty@0.7.4 install /Users/samuel/.atom/packages/script-runner/node_modules/node-pty
> node scripts/install.js

script-runner@3.0.0 /Users/samuel/.atom/packages/script-runner
├── etch@0.14.0 
├── resize-observer-polyfill@1.5.0 
├─┬ shell-environment@0.3.1 
│ └─┬ os-locale@2.1.0 
│   ├─┬ execa@0.7.0 
│   │ ├─┬ cross-spawn@5.1.0 
│   │ │ ├─┬ lru-cache@4.1.2 
│   │ │ │ ├── pseudomap@1.0.2 
│   │ │ │ └── yallist@2.1.2 
│   │ │ ├─┬ shebang-command@1.2.0 
│   │ │ │ └── shebang-regex@1.0.0 
│   │ │ └─┬ which@1.3.0 
│   │ │   └── isexe@2.0.0 
│   │ ├── get-stream@3.0.0 
│   │ ├─┬ npm-run-path@2.0.2 
│   │ │ └── path-key@2.0.1 
│   │ ├── p-finally@1.0.0 
│   │ ├── signal-exit@3.0.2 
│   │ └── strip-eof@1.0.0 
│   ├─┬ lcid@1.0.0 
│   │ └── invert-kv@1.0.0 
│   └─┬ mem@1.1.0 
│     └── mimic-fn@1.2.0 
├── shellwords@0.1.1 
├─┬ temp-write@3.4.0 
│ ├── graceful-fs@4.1.11 
│ ├── is-stream@1.1.0 
│ ├── make-dir@1.2.0 
│ ├── pify@3.0.0 
│ ├── temp-dir@1.0.0 
│ └── uuid@3.2.1 
└── xterm@3.3.0 


gyp ERR! configure error 
gyp ERR! stack Error: Python executable "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh" is v3.6.4, which is not supported by gyp.
gyp ERR! stack You can pass the --python switch to point to Python >= v2.5.0 & < 3.0.0.
gyp ERR! stack     at failPythonVersion (/Applications/Atom.app/Contents/Resources/app/apm/node_modules/node-gyp/lib/configure.js:454:14)
gyp ERR! stack     at /Applications/Atom.app/Contents/Resources/app/apm/node_modules/node-gyp/lib/configure.js:443:9
gyp ERR! stack     at ChildProcess.exithandler (child_process.js:197:7)
gyp ERR! stack     at emitTwo (events.js:106:13)
gyp ERR! stack     at ChildProcess.emit (events.js:191:7)
gyp ERR! stack     at maybeClose (internal/child_process.js:877:16)
gyp ERR! stack     at Socket.<anonymous> (internal/child_process.js:334:11)
gyp ERR! stack     at emitOne (events.js:96:13)
gyp ERR! stack     at Socket.emit (events.js:188:7)
gyp ERR! stack     at Pipe._handle.close [as _onclose] (net.js:498:12)
gyp ERR! System Darwin 17.5.0
gyp ERR! command "/Applications/Atom.app/Contents/Resources/app/apm/bin/node" "/Applications/Atom.app/Contents/Resources/app/apm/node_modules/.bin/node-gyp" "rebuild"
gyp ERR! cwd /Users/samuel/.atom/packages/script-runner/node_modules/node-pty
gyp ERR! node -v v6.9.5
gyp ERR! node-gyp -v v3.4.0
gyp ERR! not ok 
npm ERR! Darwin 17.5.0
npm ERR! argv "/Applications/Atom.app/Contents/Resources/app/apm/bin/node" "/Applications/Atom.app/Contents/Resources/app/apm/node_modules/npm/bin/npm-cli.js" "--globalconfig" "/Users/samuel/.atom/.apm/.apmrc" "--userconfig" "/Users/samuel/.atom/.apmrc" "install" "--runtime=electron" "--target=1.7.11" "--arch=x64"
npm ERR! node v6.9.5
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE

npm ERR! node-pty@0.7.4 install: `node scripts/install.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the node-pty@0.7.4 install script 'node scripts/install.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the node-pty package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node scripts/install.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs node-pty
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls node-pty
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /Users/samuel/.atom/packages/script-runner/npm-debug.log

Here is my config

 > apm config list
; cli configs
globalconfig = "/Users/samuel/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 darwin x64"
userconfig = "/Users/samuel/.atom/.apmrc"

; environment configs
node-gyp = "/Applications/Atom.app/Contents/Resources/app/apm/bin/../node_modules/.bin/node-gyp"
python = "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh"

; userconfig /Users/samuel/.atom/.apmrc

; globalconfig /Users/samuel/.atom/.apm/.apmrc
cache = "/Users/samuel/.atom/.apm"
progress = false

; node bin location = /Applications/Atom.app/Contents/Resources/app/apm/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

^_^ > npm config list
; cli configs
user-agent = "npm/3.10.10 node/v8.10.0 darwin x64"

; userconfig /Users/samuel/.npmrc
init.author.email = "samuel.williams@oriontransfer.co.nz"
init.author.name = "Samuel Williams"
init.author.url = "http://www.codeotaku.com"
python = "python2.7"

; node bin location = /opt/local/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel
; "npm config ls -l" to show all defaults.

Hmm, I'm sure I also had apm config set python=python2.7. Maybe that's why it's not working.

@ioquatix

This comment has been minimized.

Copy link

commented May 4, 2018

That's odd, maybe it's a bug in apm.

> apm config set python python2.7
> apm config list                
; cli configs
globalconfig = "/Users/samuel/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 darwin x64"
userconfig = "/Users/samuel/.atom/.apmrc"

; environment configs
node-gyp = "/Applications/Atom.app/Contents/Resources/app/apm/bin/../node_modules/.bin/node-gyp"
python = "/Applications/Atom.app/Contents/Resources/app/apm/bin/python-interceptor.sh"

; userconfig /Users/samuel/.atom/.apmrc

; globalconfig /Users/samuel/.atom/.apm/.apmrc
cache = "/Users/samuel/.atom/.apm"
progress = false

; node bin location = /Applications/Atom.app/Contents/Resources/app/apm/bin/node
; cwd = /Users/samuel/.atom/packages/script-runner
; HOME = /Users/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

Even thought I set it, it's not changed in the config!?

@raininja

This comment has been minimized.

Copy link

commented May 7, 2018

@ioquatix what does "apm config get" show?

[raijin@ishtar][~]%apm config get
; cli configs
globalconfig = "/home/raijin/.atom/.apm/.apmrc"
user-agent = "npm/3.10.10 node/v6.9.5 linux x64"
userconfig = "/home/raijin/.atom/.apmrc"

; environment configs
node-gyp = "/usr/share/atom/resources/app/apm/bin/../node_modules/.bin/node-gyp"

; userconfig /home/raijin/.atom/.apmrc
python = "python2.7"

; globalconfig /home/raijin/.atom/.apm/.apmrc
cache = "/home/raijin/.atom/.apm"
progress = false

; node bin location = /usr/share/atom/resources/app/apm/bin/node
; cwd = /home/raijin
; HOME = /home/raijin/.atom/.node-gyp
; "npm config ls -l" to show all defaults.
@ioquatix

This comment has been minimized.

Copy link

commented May 7, 2018

koyoko% apm config get
; cli configs
globalconfig = "/home/samuel/.atom/.apm/.apmrc"
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.0.0 node/v10.0.0 linux x64"
userconfig = "/home/samuel/.atom/.apmrc"

; environment configs
python = "/usr/lib/node_modules/atom-package-manager/bin/python-interceptor.sh"

; project config /home/samuel/.npmrc
strict-ssl = true

; globalconfig /home/samuel/.atom/.apm/.apmrc
cache = "/home/samuel/.atom/.apm"
progress = false

; node bin location = /usr/bin/node
; cwd = /home/samuel
; HOME = /home/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.

koyoko% apm config set python foo
koyoko% apm config get           
; cli configs
globalconfig = "/home/samuel/.atom/.apm/.apmrc"
metrics-registry = "https://registry.npmjs.org/"
scope = ""
user-agent = "npm/6.0.0 node/v10.0.0 linux x64"
userconfig = "/home/samuel/.atom/.apmrc"

; environment configs
python = "/usr/lib/node_modules/atom-package-manager/bin/python-interceptor.sh"

; project config /home/samuel/.npmrc
strict-ssl = true

; userconfig /home/samuel/.atom/.apmrc

; globalconfig /home/samuel/.atom/.apm/.apmrc
cache = "/home/samuel/.atom/.apm"
progress = false

; node bin location = /usr/bin/node
; cwd = /home/samuel
; HOME = /home/samuel/.atom/.node-gyp
; "npm config ls -l" to show all defaults.
@brennon

This comment has been minimized.

Copy link

commented May 9, 2018

It's one thing to retain support for Python 2 (though many projects are dropping it altogether). It's another thing entirely to _not _ support Python 3. Please consider this. Python 3 is a decade old, after all... 😄

@ostrolucky

This comment has been minimized.

Copy link

commented May 12, 2018

star this https://bugs.chromium.org/p/gyp/issues/detail?id=36 to help resolve this issue

@raininja

This comment has been minimized.

Copy link

commented May 12, 2018

starred, not sure why 10 years have passed and we are still dealing with this!

@Croydon

This comment has been minimized.

Copy link

commented May 13, 2018

GYP is dead. Google moved on to GN.

@ioquatix

This comment has been minimized.

Copy link

commented May 13, 2018

What is GN?

@Croydon

This comment has been minimized.

@raininja

This comment has been minimized.

Copy link

commented May 13, 2018

Yea but this thread https://groups.google.com/forum/#!topic/gyp-developer/U66xtisrYRM says it's just a matter of Dirk landing and reviewing the patch.

@raininja

This comment has been minimized.

Copy link

commented May 13, 2018

My question is this: why can't we just fork gyp and add the patch and go with that?

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented May 13, 2018

There is no reason why not. That is the way of open source. If no maintainer steps forward with a cogent plan that fits your needs, you can always hit the fork button.

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented May 13, 2018

Perhaps @refack can help to keep things progressing on this repo.

@vogt31337

This comment has been minimized.

Copy link

commented Feb 27, 2019

I don't quite understand the need for Python 2.7 (also it is clocking down less than a year).
Wouldn't it be possible to write the python code so that a python 3 interpreter is happy? I've written such code in the past which would run on both py2 and py3. And maybe this could be used for some simple use cases...

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Feb 27, 2019

@vogt31337 It is a BIG codebase. See https://github.com/refack/GYP to get a sense of how many changes were required to get it compatible with Python 3.

@megakoresh

This comment has been minimized.

Copy link

commented Mar 3, 2019

@vogt31337 It is a BIG codebase. See https://github.com/refack/GYP to get a sense of how many changes were required to get it compatible with Python 3.

I hope we can finally do away with Python 2, I am sure node-gyp is for many the last blocker remaining.

Although I do not personally quite understand why node.js really needs this wrapper around the build tool instead of letting module developers take care of it (you have to install native libraries for many native modules anyway, so its not like node-gyp and gyp itself solve the "per-library requirement" problem), but that's a different story.

@iMichka

This comment has been minimized.

Copy link

commented Apr 24, 2019

Hi. Homebrew maintainer here.

I just wanted to notify the node maintainers that we started the discussions around the Python 2 removal from the Homebrew package manager. Node is one of the last “big” packages that does not support Python 3. I would like to avoid a situation where we would need to remove node from brew. Node is in our top 3 of installed packages, with more than 4 million install events per year.

Right now we are just gathering opinions https://discourse.brew.sh/t/python-2-eol-2020/4647
No decision has been taken yet.

Options go from plain removal, additional grace period of a few months, moving node to an external tap outside of our org, or any other idea that will come up in the next weeks.

@rvagg

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

Thanks @iMichka, we have a team that's trying hard to bring Node up to Python 3 support. Unfortunately we missed out on the Node 12.0.0 window but we're acutely aware of the time pressures of continuing our Python 2 requirement. @cclauss @refack @thefourtheye and others are working hard addressing our sprawling use of Python. I'm not sure we have a firm timeframe but this is a well known problem and we all feel the pressure (and bump up against it regularly!). I would hope that we can have it resolved within the next couple of months but I can't speak with any authority on that.

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Apr 24, 2019

moving node to an external tap outside of our org

@iMichka Given that soon-to-be-released Node.js 12 will still require legacy Python, can you please provide us with a sense of the degree of difficulty required to set up and maintain an external tap? Are there other examples of external taps that your team considers to be well set up and maintained?

@scpeters

This comment has been minimized.

Copy link

commented Apr 24, 2019

I'm a homebrew maintainer, and I also maintain an external tap with bottles from private CI machines. It's super easy to make a tap and push formula updates to it. In my opinion, the bulk of the work is in getting bottles to build successfully and fixing inter-package compatibility problems. One hopefully useful option will be building homebrew bottles with something like Azure pipelines, as discussed in the following post:

@ladislas

This comment has been minimized.

Copy link

commented Apr 24, 2019

@cclauss to bounce back on what @scpeters was saying, setting up and maintaining a tap is pretty straight forward and easy.

The bottle part is a little trickier but I'm close to a fully working solution that would make it much easier for tap maintainer to distribute bottles for their taps.

I'll post here when it's ready.

@refack

This comment has been minimized.

Copy link
Member

commented Apr 24, 2019

I'm commented in several other thread that one limiting factor for getting full Python3 compatibility for GYP3, and hence node-gyp, is having a proper CI testing setup. I've submitted nodejs/build#1789 as for me setting up CI testing on the Node.js cluster is the lowest hanging fruit.

@iMichka

This comment has been minimized.

Copy link

commented Apr 24, 2019

a sense of the degree of difficulty required to **set up and maintain an _external

It is doable. The issue will be more about visibility. More than 4 million install events in a year through homebrew will mean that we need to communicate to all these people about the node removal from homebrew-core, and give instructions on how to tap the new node tap. All the install scripts from the whole industry will probably need an updated too. There will be no automatic redirection to an external tap.

I do not want to pollute this thread more with homebrew discussions though. We are not the only ones to ship node, so the issue is more global than this. Not sure this issue is the best place to discuss all this. We have not taken any decision right now, my initial message was more a gentle nudge to warn you that bigger changes may come :)

@AdamMajer

This comment has been minimized.

Copy link

commented Apr 25, 2019

We are not the only ones to ship node, so the issue is more global than this.

Indeed, it's much more global than this. There has been discussion on removing python2 from openSUSE Tumbleweed distribution at beginning of next year. For those that don't know, Tumbleweed is the rolling release distribution of openSUSE project.

https://lists.opensuse.org/opensuse-factory/2019-04/msg00229.html

on the bright fresh chilly morning of 2020-01-02 we are going to remove python2 package (that's python 2.7.* interpreter) from Factory.

@mingrammer

This comment has been minimized.

Copy link

commented Jun 5, 2019

Is this in progress?

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Jun 5, 2019

@mingrammer #1723 (comment) lists some current Python 3 incompatibilities if you have some time to work on this. We also created GYP3 nodejs/node#26620

@MattIPv4

This comment has been minimized.

Copy link
Contributor

commented Jul 6, 2019

Would it be possible to edit the opening post in this thread with a general summary of the situation with regards to the move to Python 3 due to the Python 2 EOL and drop from Brew/macOS/Linux distros? It'd be great if that could then be kept up to date so the average user can easily see an overview of the current situation and the plan going forward :)

@cclauss

This comment has been minimized.

Copy link
Contributor Author

commented Jul 7, 2019

Closing this PR as GYP seems to be Python 3 compatible:

If you find any Python 3 incompatibles in this code, please open an issue.

NOTE: nodejs/node is not yet compatible with Python 3 but an effort is ongoing there.

@superern

This comment has been minimized.

Copy link

commented Aug 3, 2019

Python 2 is starting to get removed from distributions at this point.

Correct e.g. brew is now installing/upgrading to Python 3 by default.

The following just worked for me as I already had a python2.7 executable installed via brew:

npm config set python python2.7

However, if you don't have a python2.7 executable, I suspect you can do something like:

brew install python@2
# follow the instructions printed after installing `python@2` to get python 2 on your PATH 
# e.g. echo 'export PATH="/usr/local/opt/python@2/bin:$PATH"' >> ~/.bashrc
npm config set python python2.7

Thanks! this resolves my issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.