Compiling Python

[ahgamut]

https://github.com/ahgamut/python27
https://github.com/ahgamut/cpython/tree/cosmo_py27

The assert macro needs to be changed in cosmopolitan.h to enable compilation (see #138).
Afterwards, just clone the repo and run superconfigure.

Python 2.7.18 compiled seamlessly once I figured out how autoconf worked, and what flags were being fed to the source files when running make. I'm pretty sure we can compile any C-based extensions into python.exe -- they just need to compiled/linked with Cosmopolitan, with necessary glue code added to the Python source. For example, I was able to compile SQLite into python.exe to enable the internal _sqlite module.

The compiled APE is about 4.1MB with MODE=tiny (without any of the standard modules, the interpreter alone is around 1.6MB). Most of the modules in the stdlib compile without error. The _socketmodule (required for Python's simple HTTP server) doesn't compile, as it requires the structs from netdb.h.

On Windows, the APE exits immediately because the intertpreter is unable to find the platform-specific files.
Module/getpath.c and Lib/site.py in the Python source try to use absolute paths from the prefixes provided during compilation; Editing those files to search the right locations (possibly with some zipos magic) ought to fix this.

[jart]

This is really exciting. Could you rebase your python27 repo on top of https://github.com/python/cpython so provenance is clearer and I can git diff exactly what you did? Alternatively, some kind of minimal build script showing the steps that are needed would be super helpful.

[ahgamut]

https://github.com/ahgamut/cpython/tree/cosmo_py27

Clone the repo and run superconfigure (superconfigure calls configure with the right params, then make and objcopy).

There are some minor details in the commit messages regarding what I tried to compile, etc.

[ahgamut]

Here's the sqlite fork that compiles with Cosmopolitan: https://github.com/ahgamut/sqlite/tree/cosmopolitan
Clone the repo and run superconfigure. It requires libtool and tcl8.6.

Changing the build process for SQLite was as follows:

• Added header stubs
• Created a superconfigure script to call configure with the right parameters/flags
• Changed AC_CHECK_FUNC's implementation in configure so that it doesn't error when using Cosmopolitan
• Changed Makefile target requirements to link Cosmopolitan at the end of necessary build steps
• Fixed name clashes - only one, hidden clashed with a char array
• Fixed other compilation errors - errno constants (errno constants not usable in switch-case #134) was the only error, changed the switch to an if-else

sqlite compiles without any errors (only 1 warning). I haven't figured out how to run the tests yet.

Adding sqlite to the Python build requires the above compiled sqlite and adding the below recipe to Modules/Setup.local:

# example recipe with SQLite3 
# set variables to be used in Makefile

*static*
# location of compiled https://github.com/ahgamut/sqlite
SQLITE3_DIR=../sqlite

# if there are compile-time flags with an equals sign
# set them within a string, otherwise written wrongly into the Makefile
SQLITE3_OMIT_EXTFLAG='SQLITE_OMIT_LOAD_EXTENSION=1'
SQLITE3_MOD='MODULE_NAME="sqlite3"'

# order is (module, sources, includes/defines, link locations, linked libs)
# read Modules/Setup.dist for more details
_sqlite3 _sqlite/util.c _sqlite/connection.c _sqlite/cursor.c \
    _sqlite/microprotocols.c _sqlite/cache.c  _sqlite/prepare_protocol.c \
    _sqlite/row.c _sqlite/statement.c _sqlite/module.c \
    -D$(SQLITE3_OMIT_EXTFLAG) -D$(SQLITE3_MOD) \
    -IModules/_sqlite -I$(SQLITE3_DIR) \
    -L$(SQLITE3_DIR)/.libs -lsqlite3

[ahgamut]

The python.com APE now opens on Windows!

The interpreter couldn't find the standard library because the paths were coded in as absolute paths at compile time. I changed that to use relative paths (i.e. Lib in the same directory as the interpreter). Now one can just copy python.com and the Lib folder to the same directory in a Windows machine, and the APE will find site.py and start up properly. Later it might be nice to move some of the core modules as .pyc files into a ZIP as part the APE.

Right now, running python.com yourfile.py works on Windows, but the interpreter keeps throwing syntax errors in interactive mode. It may be related to this.

Question: does Cosmopolitan handle paths (forward slash on Linux, backslash on Windows) and environment variables (separated by : on Linux, ';' on Windows) correctly?

[alisonatwork]

Your syntax error in interactive mode might be similar to the errors I experienced in trying to get various shells to run under Windows. Specifically inside Windows console when you hit enter, it will send CRLF, but the interpreter is expecting only LF for end of line, so it interprets the CR as part of the statement. You can test this by hitting Ctrl-J instead of enter. If the syntax error goes away but your cursor ends up in a funny position, then that's the problem.

For path conversion, this is done inside mkntpath.c which should be called through the standard libc functions like stat and open, but you might find you are having a different problem with PYTHONPATH, which might parse the variable using colon as separator when compiled with Cosmopolitan. For libc functions that use PATH (e.g. execlp) this is handled in commandv.c, but that won't help for PYTHONPATH, or if Python includes its own path search logic. I'm not sure if Python dynamically sets the directory and path separators at runtime or if it's compiled in, but the ideal situation would be for it to determine these at runtime the same way that Cosmopolitan does, then everything should just work.

I haven't had much time to look at this project over the past few weeks, but if I do get back to it and find any Windows-specific quirks, I'll probably post over on #117 or open a PR. I expect any solutions will be similar for shells, Python interpreter etc.

[ahgamut]

The problem is with CRLF: I just tested statements terminating with Ctrl-J on Windows, and those are accepted (I still have to press Enter to run the statement, but at least the statement runs before showing invalid syntax).

PYTHONPATH isn't an issue because I unset it before running python.com.
Directory and path separators are set at compile time (DELIM and SEP in Include/osdefs.h).

Python needs to set sys.path (locations of import-able things), sys.prefix (location of platform-independent .py/.pyc files), and sys.exec_prefix (for shared libraries) to be able to import modules correctly. There are two separate sets of functions for the path search logic:

• Modules/getpath.c contains the Unix-related stuff, and
• PC/getpathp.c contains the functions for DOS/Windows (but this is not compiled).

Both proceed similarly: check argv[0] to set the local directory, check environment variables (PATH, PYTHONPATH, PYTHONHOME, and some Windows registry stuff), try to find common locations for libraries from the directory of the executable, or finally fall back to the locations provided at compile time.

Right now, I've just changed the compile-time absolute path locations to relative paths, and it seems to work ok. Maybe after some reading I can customize Modules/getpath.c to have a IsWindows() check and change everything accordingly.

[alisonatwork]

It's really annoying that POSIX doesn't define a function to search PATH, it seems every shell just reimplements it for itself, and then libc has its own way again for execlp etc. I think something that might make our lives easier is publishing a Cosmopolitan-blessed path searcher function, so given an environment variable name (or buf with the value already in it), search each path for a file underneath it in a platform-agnostic way. Something like the SearchPath function in commandv.c, but which works for other variables, and where you can toggle search for executables or just search for any file. That might be able to replace some of the functions that Python, ash etc are trying to use to find the right file to load (or autocomplete, or whatever).

The CRLF thing is a trickier problem. Something you can try is using mintty (easiest way is from Git Bash) as your "terminal" instead of Windows console. I'm not sure, but that might avoid the generation of a CRLF when hitting enter, which would at least be a temporary workaround. Solving the problem inside console is more challenging. Personally I don't see it as very useful that carriage return is parsed as a non-whitespace token, even on UNIX. It seems to me the cleanest solution would be for Python to ignore CR, or treat it the same as a trailing space. That's the approach I went with trying to get ash to work, but I'm not sure if it will have unintended consequences for files with binary data in them.

[alisonatwork]

Maybe we could "polyfill" this one for UNIX: https://docs.microsoft.com/en-us/windows/win32/api/processenv/nf-processenv-searchpatha It could perhaps be used to handle finding things in the APE's ZIP filesystem too.

[ahgamut]

A quick list of the internal modules that can't be compiled yet (full list in the repo README):

• syslog now compiles with the latest commit
• _sqlite3 compiles if I have libsqlite3.a
• bz2 compiles if I have libbz2.a
• mmap -- mremap required an additional void * parameter (new_address), currently passing NULL
• _ctypes -- required libffi.a which compiles if you disable all pthread and mntent-related stuff
• readline compiles with some modifications to libreadline.a and libtermcap.a, but not particularly useful
• _locale -- linker error searching for strxfrm
• _socket -- requires hostent, protoent and other structs from netdb.h
• crypt -- requires -lcrypt
• _hashlib and _ssl require linking with OpenSSL (which requires netdb.h to compile)

_multiprocessing requires _save, which is a variable that is part of Py_BEGIN_ALLOW_THREADS.
Of the remaining modules, maybe dbm/gdbm would be useful to have.

[ahgamut]

The Python tokenizer now ignores CR when reading input. No more syntax errors when the APE runs in interactive mode on Windows!

[niutech]

I have built the APE binary of Python 2.7, you can download it here.

[jart]

We could generalize the commandv API but I suspect one of the reasons why POSIX doesn't do that already is that PATH searching is such an expensive operation (in terms of system calls and disk seeks) and shells usually implement it on their own because they're able to perform local optimizations (like memoization) that the C library isn't able to do. For example, sometimes when using the bash shell, I'll need to occasionally run hash -r to let it know to recompute PATH after it's been changed.

[ahgamut]

Maybe just skip looking through environment variables when starting up python.com? The interpreter looks through PATH (and also PYTHONPATH, PYTHONHOME) only because it needs to find the necessary standard library modules and .so/.dll shared libs. But that can be changed entirely or skipped: in this commit I've commented out the search for PYTHONPATH and PYTHONHOME, and I'm already using relative paths for the directories.

[niutech]

As a workaround for Cosmopolitan Python, you can build the APE version of the latest Wasm3 and run Rust Python WebAssembly interpreter in it:

$ ./wasm3.com --stack-size 1000000 rustpython.wasm 
Welcome to the magnificent Rust Python 0.1.1 interpreter 😱 🖖
>>>>>