Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: failed to solve: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1 #411

Closed
ibmgeniuz opened this issue Apr 17, 2024 · 7 comments

Comments

@ibmgeniuz
Copy link

I tried to set up gl-testing on my machine following this documentation https://blockstream.github.io/greenlight/tutorials/testing/. I always got stuck with the error below. I did not make any modifications. Any idea why this may be the case?

Steps to reproduce

  • Clone the github repository
  • Run make docker-image

Computer Specs

  • M2 MacBook Air
  • Docker 4.29.0
192.8   error: failed to run custom build command for `cln-grpc v0.1.8`
192.8
192.8   Caused by:
192.8     process didn't exit successfully: `/repo/target/release/build/cln-grpc-ef2f5f5edc573bda/build-script-build` (exit status: 101)
192.8     --- stdout
192.8     cargo:rerun-if-changed=proto/node.proto
192.8     cargo:rerun-if-changed=proto
192.8
192.8     --- stderr
192.8     thread 'main' panicked at /opt/cargo/registry/src/index.crates.io-6f17d22bba15001f/cln-grpc-0.1.8/build.rs:7:10:
192.8     called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "protoc failed: rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2\n " }

192.8     note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
192.8   warning: build failed, waiting for other jobs to finish...
192.8   💥 maturin failed
192.8     Caused by: Failed to build a native library through cargo
192.8     Caused by: Cargo build finished with "exit status: 101": `"cargo" "rustc" "--message-format" "json-render-diagnostics" "--manifest-path" "/repo/libs/gl-client-py/Cargo.toml" "--release" "--lib"`
192.8   Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/tmp/tmpompj7ofu/.venv/bin/python', '--compatibility', 'off', '--editable'] returned non-zero exit status 1
192.8
192.8
192.8   at /tmp/poetry/venv/lib/python3.8/site-packages/poetry/installation/chef.py:164 in _prepare
192.8       160│
192.8       161│                 error = ChefBuildError("\n\n".join(message_parts))
192.8       162│
192.8       163│             if error is not None:
192.8     → 164│                 raise error from None
192.8       165│
192.8       166│             return path
192.8       167│
192.8       168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
192.8
192.8 Note: This error originates from the build backend, and is likely not a problem with poetry but with gl-client (0.1.11 /repo/libs/gl-client-py) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 --editable "/repo/libs/gl-client-py"'.
192.8
------
Dockerfile:274
--------------------
 272 |
 273 |     # Add the remaining repositories to the python-path
 274 | >>> RUN poetry install
 275 |     RUN chmod -R a+rw /tmp/venv
 276 |
--------------------

ERROR: failed to solve: process "/bin/sh -c poetry install" did not complete successfully: exit code: 1

View build details: docker-desktop://dashboard/build/desktop-linux/desktop-linux/ql3hsngo5lnht6va2vfdbnr8l
make: *** [docker-image] Error 1
@ErikDeSmedt
Copy link
Collaborator

ErikDeSmedt commented Apr 18, 2024

I suspect the issue is that one of our dependencies does not support the M2 processor architecture.
We download a binary for protoc that can only be used on x86_64.

As a work-around you could try and use a version that is compiled for Mac M2.
You can do this by editin ./docker/gl-testing/Dockerfile in line 163. You replace the url to a binary that works on your machine.

I don't have a Mac with M2-architecture so I can't test locally. Let me know if you need more details or help

@cdecker
Copy link
Collaborator

cdecker commented Apr 18, 2024

Hm, that's very strange though, it talks about rosetta which is the CPU virtualization framework used to emulate x86 on arm64 and exists solely for Mac OS. This is inside of Docker however, which is an emulated intel linux machine iirc, so how can it end up trying to execute an intel binary inside the docker image, with the emulator on the host?

I'm confused tbh. But maybe it is actually emulating an arm64 linux machine? That would explain the mismatch, but not how the host emulator is involved.

@ibmgeniuz
Copy link
Author

@ErikDeSmedt Are you talking about the releases on this page https://github.com/protocolbuffers/protobuf/releases ?
What is in the Dockerfile seem to be pretty outdated compared to what I see on that page. https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip but I see v26.1 . Am I on the right page?

Anyway I changed the linux-x86_64 part to linux-aarch_64 . It's been running for more than 1200 secs at 54/56. So far it has not given the error (fingers crossed) but I don't see progress yet. I will let you know how it goes.

@cdecker I was also thinking the same. I don't have rosetta on my Mac. However, if the whole idea of using docker is to ensure that we don't have to install dependencies on our computer, then it's surprising to me that I would need to install rosetta. I've held off installing rosetta on my computer for a very long time so it will be a real bummer if I would need to install it before this can work.

@ibmgeniuz
Copy link
Author

@ErikDeSmedt @cdecker the command make docker-image completed successfully after I replaced https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-x86_64.zip with https://github.com/protocolbuffers/protobuf/releases/download/v3.19.3/protoc-3.19.3-linux-aarch_64.zip . I went ahead to run

  • make docker-shell
  • exited the shell.
  • created my-first-test.py at the root of the repository gl-testing-tutorial with contents same as in the tutorial
  • executed make build-self after make docker-shell
  • Then I ran pytest -v -s my-first-test.py but got the error below.

Any idea why this is the case? Do I need to create a separate issue?

―――――――――――――――――――――――――――――― ERROR at setup of test_invoice_payment ――――――――――――――――――――――――――――――

directory = PosixPath('/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing')
teardown_checks = <pyln.testing.fixtures.TeardownErrors object at 0xffff836377c0>

    @pytest.fixture
    def bitcoind(directory, teardown_checks):
        chaind = network_daemons[env('TEST_NETWORK', 'regtest')]
        bitcoind = chaind(bitcoin_dir=directory)

        try:
>           bitcoind.start()

/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:132:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:437: in start
    self.wait_for_log("Done loading", timeout=TIMEOUT)
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:347: in wait_for_log
    return self.wait_for_logs([regex], timeout)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyln.testing.utils.BitcoinD object at 0xffff83637670>, regexs = ['Done loading']
timeout = 60

    def wait_for_logs(self, regexs, timeout=TIMEOUT):
        """Look for `regexs` in the logs.

        The logs contain tailed stdout of the process. We look for each regex
        in `regexs`, starting from `logsearch_start` which normally is the
        position of the last found entry of a previous wait-for logs call.
        The ordering inside `regexs` doesn't matter.

        We fail if the timeout is exceeded or if the underlying process
        exits before all the `regexs` were found.

        If timeout is None, no time-out is applied.
        """
        logging.debug("Waiting for {} in the logs".format(regexs))
        exs = [re.compile(r) for r in regexs]
        start_time = time.time()
        while True:
            if self.logsearch_start >= len(self.logs):
                if not self.logs_catchup():
                    time.sleep(0.25)

                if timeout is not None and time.time() > start_time + timeout:
                    print("Time-out: can't find {} in logs".format(exs))
                    for r in exs:
                        if self.is_in_log(r):
                            print("({} was previously in logs!)".format(r))
>                   raise TimeoutError('Unable to find "{}" in logs.'.format(exs))
E                   TimeoutError: Unable to find "[re.compile('Done loading')]" in logs.

/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:328: TimeoutError

During handling of the above exception, another exception occurred:

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

/tmp/venv/lib/python3.8/site-packages/flaky/flaky_pytest_plugin.py:146:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/tmp/venv/lib/python3.8/site-packages/pyln/testing/fixtures.py:134: in bitcoind
    bitcoind.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:448: in stop
    self.rpc.stop()
/tmp/venv/lib/python3.8/site-packages/pyln/testing/utils.py:374: in f
    res = proxy._call(name, *args)
/tmp/venv/lib/python3.8/site-packages/bitcoin/rpc.py:244: in _call
    self.__conn.request('POST', self.__url.path, postdata, headers)
/usr/lib/python3.8/http/client.py:1256: in request
    self._send_request(method, url, body, headers, encode_chunked)
/usr/lib/python3.8/http/client.py:1302: in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1251: in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
/usr/lib/python3.8/http/client.py:1011: in _send_output
    self.send(msg)
/usr/lib/python3.8/http/client.py:951: in send
    self.connect()
/usr/lib/python3.8/http/client.py:922: in connect
    self.sock = self._create_connection(
/usr/lib/python3.8/socket.py:808: in create_connection
    raise err
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

address = ('localhost', 45091), timeout = 30, source_address = None

    def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT,
                          source_address=None):
        """Connect to *address* and return the socket object.

        Convenience function.  Connect to *address* (a 2-tuple ``(host,
        port)``) and return the socket object.  Passing the optional
        *timeout* parameter will set the timeout on the socket instance
        before attempting to connect.  If no *timeout* is supplied, the
        global default timeout setting returned by :func:`getdefaulttimeout`
        is used.  If *source_address* is set it must be a tuple of (host, port)
        for the socket to bind as a source address before making the connection.
        A host of '' or port 0 tells the OS to use the default.
        """

        host, port = address
        err = None
        for res in getaddrinfo(host, port, 0, SOCK_STREAM):
            af, socktype, proto, canonname, sa = res
            sock = None
            try:
                sock = socket(af, socktype, proto)
                if timeout is not _GLOBAL_DEFAULT_TIMEOUT:
                    sock.settimeout(timeout)
                if source_address:
                    sock.bind(source_address)
>               sock.connect(sa)
E               ConnectionRefusedError: [Errno 111] Connection refused

/usr/lib/python3.8/socket.py:796: ConnectionRefusedError
---------------------------------------- Captured log setup ----------------------------------------
DEBUG    root:utils.py:225 Starting 'bitcoind -datadir=/tmp/ltests-vbm295qu/test_invoice_payment_1/gl-testing -printtoconsole -server -logtimestamps -nolisten -txindex -nowallet -addresstype=bech32 -debug=mempool -debug=mempoolrej -debug=rpc -debug=validation'
DEBUG    root:utils.py:315 Waiting for ['Done loading'] in the logs
DEBUG    root:utils.py:285 Did not find 're.compile('Done loading')' in logs
DEBUG    root:utils.py:370 Calling stop with arguments ()
                                                                                     100% ██████████
========================================= warnings summary =========================================
../tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7
  /tmp/venv/lib/python3.8/site-packages/cheroot/__init__.py:7: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
    import pkg_resources

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================== short test summary info ======================================
FAILED my-first-test.py::test_invoice_payment - ConnectionRefusedError: [Errno 111] Connection refused

Results (61.01s (0:01:01)):
       1 error

@ErikDeSmedt
Copy link
Collaborator

ErikDeSmedt commented Apr 19, 2024 via email

@ibmgeniuz
Copy link
Author

Alright. Thank you. I will see if I can hack my way around it. Whatever be the case, I will let you know. I will keep this issue opened in case we reach some solutions somehow.

I have no experience building on Mac. However, I think you might be in for a rough ride. Currently we only support x86_64. The problem appears to be that bitcoind doesnt't start. You could try to download a bitcoind that is suitable for your architecture and try again. The bigger problem is that you will also need a wide range of Core Lightning versions. For those is no M2 version available and building them yourself would be very tricky.

@cdecker
Copy link
Collaborator

cdecker commented May 14, 2024

Closing this one as it is more related to the dev setup being messed up between the various architectures (thanks Rosetta) and not with the software in this repo.

@cdecker cdecker closed this as completed May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants