Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make tools/configure.py work with Python 3.x #1794

Closed
ghost opened this issue Nov 5, 2017 · 23 comments
Closed

Make tools/configure.py work with Python 3.x #1794

ghost opened this issue Nov 5, 2017 · 23 comments

Comments

@ghost
Copy link

ghost commented Nov 5, 2017

Currently tools/configure.py fails with an error when run with Python 3. It makes things complicated when your build system already uses other scripts done in Python 3. In the perfect scenario both flavors, Py2 and Py3, would be supported.

@svaarala
Copy link
Owner

svaarala commented Nov 5, 2017

I tried doing that conversion a while back - it's really painful for the same Python source to work in both environments.

@svaarala
Copy link
Owner

svaarala commented Nov 5, 2017

One option I've considered is moving away entirely from Python because of the Python2/Python3 rift (which is awkward for everyone involved). A logical candidate would be Node.js which is also well supported on major platforms. Another approach would be to be self-hosting, i.e. compile a command line tool with the necessary file I/O bindings for doing the configuration step. But no actions on this so far.

@fatcerberus
Copy link
Contributor

Node.js is nice but seems like a very heavy dependency, even heavier than Python. I rather like the idea of self-hosting, personally. I imagine that would be a very large undertaking, though.

@fatcerberus
Copy link
Contributor

Hmm... Is there any benefit nowadays to retaining Python 2 support at all? I feel like we could upgrade the build tools to Python 3-only and it wouldn't be an issue, as long as there's a way to ensure you run the correct one on systems with both installed.

@svaarala
Copy link
Owner

svaarala commented Nov 6, 2017

Other than breaking existing builds I think Python 3 only might be a viable option today. So that change could be made for Duktape 3.x, but I wouldn't want to make it for Duktape 2.x because existing builds would quite likely break.

Self-hosting wouldn't be a large undertaking as far as the build script functionality is concerned. The main concern would be a build dependency to maybe a different Duktape version to run the configuration tooling. It'd be easy if there was a pre-packaged command line binary for major platforms, but lacking that it seems awkward to use (and document in an understandable fashion). So if this approach was taken, it'd be good to release a pre-packaged binary tool for the most common platforms.

@fatcerberus
Copy link
Contributor

For what it’s worth I just found out Python 2 won’t be maintained at all after 2020, so that’s probably a good reason to migrate sooner rather than later.

@svaarala
Copy link
Owner

That might pose a risk for availability but for the tooling itself it's not critical that the implementation is maintained (e.g. for security issues), because it's so far removed from actual execution. But certainly availability will eventually become worse closer to 2020. But even so, I don't think it's an immediate concern for at least a few years.

@svaarala
Copy link
Owner

Given a 1-2 year migration window, I think I'll probably start by looking at non-Python alternatives first. After all, maybe there's a hard python3-to-python4 transition down the road and I don't want to go through that again ;)

@svaarala
Copy link
Owner

If the tooling remains Python then one approach is to look at the automatic 2-to-3 conversions. The result may require a small fixup, and it might be feasible to maintain support for both for some time with that approach, then eventually switch to Python 3 only.

@fatcerberus
Copy link
Contributor

After all, maybe there's a hard python3-to-python4 transition down the road and I don't want to go through that again ;)

Yeah...

That’s what’s really nice about the Ecmascript process, backwards compatibility is maintained so a lot of code that was written even 2 decades ago still runs without modification in a modern JS engine. Sometimes we lose out on new features (like global) because of it, but the backward compatibility is really nice. No Python-like fragmentation. :)

@TheBrokenRail
Copy link
Contributor

Being the last of the 2.x series, 2.7 will have an extended period of maintenance. Specifically, 2.7 will receive bugfix support until January 1, 2020. After the last release, 2.7 will receive no support.
https://www.python.org/dev/peps/pep-0373/#maintenance-releases

Any updates on updating to Python 3 or moving way from Python?

@svaarala
Copy link
Owner

No updates at present. The dependency on python2 is not very problematic in my opinion; python2 is needed to create sources and headers for a custom configuration (not for compilation itself) and as such is easy to e.g. dockerize. But eventually some solution will be needed.

@svaarala
Copy link
Owner

svaarala commented Jun 2, 2019

I've been giving this a little thought and if going with a non-Python approach this approach might be nice:

Rewrite the tooling and other utility stuff into Javascript modules. Use pure Javascript YAML and XML libraries (which exist and seem to work well), so that native dependencies are minimized. In effect one would only rely on:

  • The ability to parse command line arguments
  • Minimal file I/O primitives, e.g. listDir(), readFile(), and writeFile()

This tooling could be run directly with Node.js which is available for major platforms, and is fast which is nice for Unicode processing and such. But because the native binding assumptions are minimal, it would also be easy to run it with Duktape or any other ES5.1 engine (transpiled if necessary) so that the tooling could also be made available on (almost) all targets where Duktape itself runs (which currently isn't possible).

I've experimented a bit with this and it seems straightforward. The pure JS bindings I've used so far are js-yaml and from-xml which seem to fit the purpose well.

Rewriting the tooling into JS would also provide an opportunity to clean up the tooling in the process.

@Ozaq
Copy link

Ozaq commented Feb 6, 2020

@svaarala I am currently in the situation that I am maintaining a build environment that only depends on python2 for duktape, I very much would like to move away from that. Would you be open for a PR that ports to python3? I do not want to get started on anything if you have set you mind on moving to JS.

Oh and btw duktape is cool stuff ;)

@svaarala
Copy link
Owner

I think moving to JS makes most sense, and I actually already have most of the tooling already rewritten in JS. It's waiting to be merged when I get a chance to work on it for a few days :)

The JS tooling works on Node.js but also on Duktape itself (not just in theory but in practice too :), making it possible to run the tooling on any target supporting Duktape. To run the tooling on Duktape one needs to bootstrap Duktape with a few additional native bindings (like a wrapper for system(), access to command line arguments, etc).

@strega-nil
Copy link

I'm also interested in this, since vcpkg only requires python2, I believe, for this exact library 😛

strega-nil added a commit to strega-nil/vcpkg that referenced this issue Feb 12, 2020
brew no longer contains python2, since it's EOL, and duktape requires that for now

see svaarala/duktape#1794
ras0219-msft pushed a commit to microsoft/vcpkg that referenced this issue Feb 12, 2020
brew no longer contains python2, since it's EOL, and duktape requires that for now

see svaarala/duktape#1794
@Ozaq
Copy link

Ozaq commented Feb 12, 2020

@svaarala alright, thanks for the update. Much appreciated.

KineticTheory pushed a commit to KineticTheory/vcpkg that referenced this issue Feb 17, 2020
brew no longer contains python2, since it's EOL, and duktape requires that for now

see svaarala/duktape#1794
@ethindp
Copy link

ethindp commented Nov 10, 2020

Any update on this? I just tried creating a custom config for Duktape and I can't because I use Python 3.9 only. Tried updating it and ran into a ton of tracebacks.

@svaarala
Copy link
Owner

svaarala commented Nov 11, 2020

The JS tooling is in master but it's not yet used by default Makefiles etc. But you can use it as follows:

$ make build/duktool.js
[...]
$ nodejs build/duktool.js configure --help
[...]
$ nodejs build/duktool.js configure --output-directory /tmp/test --source-directory src-input/
[...]
$ ls -l /tmp/test
total 4888
-rw-r--r-- 1 duktape duktape  103084 Nov 11 17:14 duk_config.h
-rw-r--r-- 1 duktape duktape   25321 Nov 11 17:14 duk_source_meta.json
-rw-r--r-- 1 duktape duktape 4791986 Nov 11 17:14 duktape.c
-rw-r--r-- 1 duktape duktape   75281 Nov 11 17:14 duktape.h

Later I'll merge a change that changes configure.py into a compatible wrapper that invokes the JS code (with a deprecation warning).

@jberryman
Copy link

Oh neat I didn't notice the js implementation until reading through the thread and will try it out, thanks for doing that!

Fwiw debian testing no longer ships anything except the python2 interpreter itself, so it feels truly like the late twilight of that language...

@svaarala svaarala closed this as completed Apr 8, 2022
@saghul
Copy link
Contributor

saghul commented Apr 10, 2022

Was the script migrated to Python 3 or JS?

@svaarala
Copy link
Owner

It's Node.js based now, in https://github.com/svaarala/duktape/tree/master/src-tools. It will eventually be self-hostable with Duktape too.

@janisozaur
Copy link

Is node-based tool the preferred way now? On 2.7.0 already? https://wiki.duktape.org/configuring2x still lists python util only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants