Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make start failing #9

Closed
kenz-gelsoft opened this issue Apr 11, 2023 · 7 comments
Closed

make start failing #9

kenz-gelsoft opened this issue Apr 11, 2023 · 7 comments

Comments

@kenz-gelsoft
Copy link
Owner

~/src/firefox-102.9.0/obj-x86_64-unknown-haiku> make
make[1]: Entering directory '/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku'
make recurse_pre-export
make[2]: Entering directory '/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku'
/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku/_virtualenvs/build/non-packaged/bin/python -m mozbuild.action.process_install_manifest --track install_dist_include.track dist/include _build_manifests/install/dist_include
/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku/_virtualenvs/build/non-packaged/bin/python: Error while finding module specification for 'mozbuild.action.process_install_manifest' (ModuleNotFoundError: No module named 'mozbuild')
Makefile:94: recipe for target 'install-dist/include' failed
make[2]: *** [install-dist/include] Error 1
make[2]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku'
/boot/home/src/firefox-102.9.0/config/recurse.mk:32: recipe for target 'pre-export' failed
make[1]: *** [pre-export] Error 2
make[1]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-x86_64-unknown-haiku'
/boot/home/src/firefox-102.9.0/config/rules.mk:361: recipe for target 'default' failed
make: *** [default] Error 2
~/src/firefox-102.9.0/obj-x86_64-unknown-haiku> 

we may missed some steps to build with old-configure

@kenz-gelsoft
Copy link
Owner Author

@kenz-gelsoft
Copy link
Owner Author

it seems virtualenv failed to resolve python path again

@kenz-gelsoft
Copy link
Owner Author

kenz-gelsoft commented Apr 12, 2023

https://peps.python.org/pep-0405/#specification says:

Virtualenv must copy the binary in order to provide isolation, as Python dereferences a symlinked executable before searching for sys.prefix.

but I saw virtualenv python binaries were symlinks.

This PEP proposes to add a new first step to this search. If a pyvenv.cfg file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form key = value. If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.

In this case, prefix-finding continues as normal using the value of the home key as the effective Python binary location, which finds the prefix of the base installation. sys.base_prefix is set to this value, while sys.prefix is set to the directory containing pyvenv.cfg.

(If pyvenv.cfg is not found or does not contain the home key, prefix-finding continues normally, and sys.prefix will be equal to sys.base_prefix.)

Okay, it doesn't work well because pyvenv.cfg was created outside of non-packaged subdirectory. This doesn't meet above criteria.

@kenz-gelsoft
Copy link
Owner Author

kenz-gelsoft commented Apr 13, 2023

We had to fix up virtualenv created by Haiku's python.

diff --git a/python/mach/mach/site.py b/python/mach/mach/site.py
index 7065f60020..47763d532d 100644
--- a/python/mach/mach/site.py
+++ b/python/mach/mach/site.py
@@ -1184,6 +1184,12 @@ def _create_venv_with_pthfile(
         ]
     )
 
+    for dir in ("bin", "lib"):
+        src = os.path.join(virtualenv_root, "non-packaged", dir)
+        dst = os.path.join(virtualenv_root, dir)
+        shutil.rmtree(dst, ignore_errors=True)
+        os.symlink(src, dst, target_is_directory=True)
+
     platlib_site_packages_dir = target_venv.resolve_sysconfig_packages_path("platlib")
     pthfile_contents = "\n".join(pthfile_lines)
     with open(os.path.join(platlib_site_packages_dir, PTH_FILENAME), "w") as f:

now build proceeds

make[3]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-a/dom/bindings/test'
./cbindgen-metadata.json.stub
/boot/home/src/firefox-102.9.0/obj-a/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /boot/home/src/firefox-102.9.0/build/RunCbindgen.py generate_metadata config/cbindgen-metadata.json config/.deps/cbindgen-metadata.json.pp config/.deps/cbindgen-metadata.json.stub .cargo/config

thread 'main' panicked at 'unexpected error from isatty: 0', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/rustix-0.36.3/src/backend/libc/termios/syscalls.rs:141:20
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

BUILDTASK {"argv": ["/boot/home/src/firefox-102.9.0/python/mozbuild/mozbuild/action/file_generate.py", "/boot/home/src/firefox-102.9.0/build/RunCbindgen.py", "generate_metadata", "config/cbindgen-metadata.json", "config/.deps/cbindgen-metadata.json.pp", "config/.deps/cbindgen-metadata.json.stub", ".cargo/config"], "start": 1681399491.080572, "end": 1681399491.1969259, "context": null}
backend.mk:114: recipe for target 'config/.deps/cbindgen-metadata.json.stub' failed
make[2]: *** [config/.deps/cbindgen-metadata.json.stub] Error 101
make[2]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-a'
/boot/home/src/firefox-102.9.0/config/recurse.mk:32: recipe for target 'export' failed
make[1]: *** [export] Error 2
make[1]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-a'
/boot/home/src/firefox-102.9.0/config/rules.mk:361: recipe for target 'default' failed
make: *** [default] Error 2

@kenz-gelsoft
Copy link
Owner Author

kenz-gelsoft commented Apr 13, 2023

Worked around by running cargo through pty

diff --git a/build/RunCbindgen.py b/build/RunCbindgen.py
index 4c5bc8ce6e..d506c9e61b 100644
--- a/build/RunCbindgen.py
+++ b/build/RunCbindgen.py
@@ -8,6 +8,7 @@ import mozpack.path as mozpath
 import os
 import six
 import subprocess
+import pty
 import pytoml
 
 
@@ -41,7 +42,12 @@ def _run_process(args):
 
 
 def generate_metadata(output, cargo_config):
-    stdout, returncode = _run_process(
+    # stdout, returncode = _run_process(
+    def read(fd):
+        data = os.read(fd, 1)
+        output.write(data)
+        return data
+    pty.spawn(
         [
             buildconfig.substs["CARGO"],
             "metadata",
@@ -52,12 +58,13 @@ def generate_metadata(output, cargo_config):
             "--manifest-path",
             CARGO_TOML,
         ]
-    )
+    # )
+    , read)
 
-    if returncode != 0:
-        return returncode
+    # if returncode != 0:
+    #     return returncode
 
-    output.write(stdout)
+    # output.write(stdout)
 
     # This is not quite accurate, but cbindgen only cares about a subset of the
     # data which, when changed, causes these files to change.
@@ -65,7 +72,12 @@ def generate_metadata(output, cargo_config):
 
 
 def generate(output, metadata_path, cbindgen_crate_path, *in_tree_dependencies):
-    stdout, returncode = _run_process(
+    # stdout, returncode = _run_process(
+    def read(fd):
+        data = os.read(fd, 1)
+        output.write(data)
+        return data
+    pty.spawn(
         [
             buildconfig.substs["CBINDGEN"],
             buildconfig.topsrcdir,
@@ -77,12 +89,13 @@ def generate(output, metadata_path, cbindgen_crate_path, *in_tree_dependencies):
             metadata_path,
             "--cpp-compat",
         ]
-    )
+    # )
+    , read)
 
-    if returncode != 0:
-        return returncode
+    # if returncode != 0:
+    #     return returncode
 
-    output.write(stdout)
+    # output.write(stdout)
 
     deps = set()
     deps.add(CARGO_LOCK)

references:

@kenz-gelsoft
Copy link
Owner Author

diff --git a/build/RunCbindgen.py b/build/RunCbindgen.py
index 4c5bc8ce6e..7130922811 100644
--- a/build/RunCbindgen.py
+++ b/build/RunCbindgen.py
@@ -4,10 +4,13 @@
 
 from __future__ import print_function
 import buildconfig
+import errno
 import mozpack.path as mozpath
 import os
+import select
 import six
 import subprocess
+import pty
 import pytoml
 
 
@@ -29,18 +32,54 @@ def _run_process(args):
     env["CARGO"] = str(buildconfig.substs["CARGO"])
     env["RUSTC"] = str(buildconfig.substs["RUSTC"])
 
-    p = subprocess.Popen(args, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
-
-    stdout, stderr = p.communicate()
-    stdout = six.ensure_text(stdout)
-    stderr = six.ensure_text(stderr)
+    #p = subprocess.Popen(args, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    #stdout, stderr = p.communicate()
+    #stdout = six.ensure_text(stdout)
+    #stderr = six.ensure_text(stderr)
+    mo, so = pty.openpty()
+    me, se = pty.openpty()
+    p = subprocess.Popen(args, env=env, stdout=so, stderr=se, close_fds=True)
+    for fd in [so, se]:
+        os.close(fd)
+    
+    timeout = 0.04 # seconds
+    readable = [mo, me]
+    result = {mo: b'', me: b''}
+    try:
+    try:
+        while readable:
+            ready, _, _ = select.select(readable, [], [], timeout)
+            for fd in ready:
+                try:
+                    data = os.read(fd, 512)
+                except OSError as e:
+                    if e.errno != errno.EIO:
+                        raise
+                    # EIO means EOF on some systems
+                    readable.remove(fd)
+                else:
+                    if not data: # EOF
+                        readable.remove(fd)
+                    result[fd] += data
+    finally:
+        for fd in [mo, me]:
+            os.close(fd)
+        if p.poll() is None:
+            p.kill()
+        p.wait()
+    
+    stdout = six.ensure_text(result[mo])
+    stderr = six.ensure_text(result[me])
     if p.returncode != 0:
         print(stdout)
         print(stderr)
     return (stdout, p.returncode)
 
-
 def generate_metadata(output, cargo_config):
+    #def read(fd):
+    #    data = os.read(fd, 1024)
+    #    output.write(data)
+    #    return data
+    #pty.spawn(
     stdout, returncode = _run_process(
         [
             buildconfig.substs["CARGO"],
@@ -53,6 +92,7 @@ def generate_metadata(output, cargo_config):

@kenz-gelsoft
Copy link
Owner Author

kenz-gelsoft commented Apr 21, 2023

work around is not required now with patched cargo.

#17 (comment)

make[3]: Leaving directory '/boot/home/src/firefox-102.9.0/obj-a/dom/bindings'
./cbindgen-metadata.json.stub
/boot/home/src/firefox-102.9.0/obj-a/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /boot/home/src/firefox-102.9.0/build/RunCbindgen.py generate_metadata config/cbindgen-metadata.json config/.deps/cbindgen-metadata.json.pp config/.deps/cbindgen-metadata.json.stub .cargo/config
BUILDTASK {"argv": ["/boot/home/src/firefox-102.9.0/python/mozbuild/mozbuild/action/file_generate.py", "/boot/home/src/firefox-102.9.0/build/RunCbindgen.py", "generate_metadata", "config/cbindgen-metadata.json", "config/.deps/cbindgen-metadata.json.pp", "config/.deps/cbindgen-metadata.json.stub", ".cargo/config"], "start": 1682116372.160727, "end": 1682116383.869293, "context": null}
./audioipc2_client_ffi_generated.h.stub
/boot/home/src/firefox-102.9.0/obj-a/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /boot/home/src/firefox-102.9.0/build/RunCbindgen.py generate dom/media/audioipc2_client_ffi_generated.h dom/media/.deps/audioipc2_client_ffi_generated.h.pp dom/media/.deps/audioipc2_client_ffi_generated.h.stub config/cbindgen-metadata.json /boot/home/src/firefox-102.9.0/third_party/rust/audioipc2-client
BUILDTASK {"argv": ["/boot/home/src/firefox-102.9.0/python/mozbuild/mozbuild/action/file_generate.py", "/boot/home/src/firefox-102.9.0/build/RunCbindgen.py", "generate", "dom/media/audioipc2_client_ffi_generated.h", "dom/media/.deps/audioipc2_client_ffi_generated.h.pp", "dom/media/.deps/audioipc2_client_ffi_generated.h.stub", "config/cbindgen-metadata.json", "/boot/home/src/firefox-102.9.0/third_party/rust/audioipc2-client"], "start": 1682116385.0819101, "end": 1682116385.3996649, "context": null}
./audioipc2_server_ffi_generated.h.stub
/boot/home/src/firefox-102.9.0/obj-a/_virtualenvs/build/bin/python -m mozbuild.action.file_generate /boot/home/src/firefox-102.9.0/build/RunCbindgen.py generate dom/media/audioipc2_server_ffi_generated.h dom/media/.deps/audioipc2_server_ffi_generated.h.pp dom/media/.deps/audioipc2_server_ffi_generated.h.stub config/cbindgen-metadata.json /boot/home/src/firefox-102.9.0/third_party/rust/audioipc2-server
BUILDTASK {"argv": ["/boot/home/src/firefox-102.9.0/python/mozbuild/mozbuild/action/file_generate.py", "/boot/home/src/firefox-102.9.0/build/RunCbindgen.py", "generate", "dom/media/audioipc2_server_ffi_generated.h", "dom/media/.deps/audioipc2_server_ffi_generated.h.pp", "dom/media/.deps/audioipc2_server_ffi_generated.h.stub", "config/cbindgen-metadata.json", "/boot/home/src/firefox-102.9.0/third_party/rust/audioipc2-server"], "start": 1682116386.596372, "end": 1682116387.003677, "context": null}
./audioipc_client_ffi_generated.h.stub

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant