bug: devcontainer is missing GDAL #9011

anentropic · 2024-04-18T17:42:09Z

What happened?

I checked out the repo and opened it in VS Code. It found a .devcontainer config and prompted me to open in a container... I thought this might be easier than messing around with conda etc

But it doesn't work. The updateContent.sh script fails trying to pip install dependencies with an error:

Failed to get options via gdal-config: [Errno 2] No such file or directory: 'gdal-config'
      CRITICAL:root:A GDAL API version must be specified. Provide a path to gdal-config using a GDAL_CONFIG environment variable or use a GDAL_VERSION environment variable.

Presumably gdal is not installed in the mcr.microsoft.com/vscode/devcontainers/python:3.11 base image

By adding "remoteUser": "vscode" to the devcontainer.json I was able to sudo apt-get install libgdal-dev

This got further but hit another problem:

Defaulting to user installation because normal site-packages is not writeable
Obtaining file:///workspaces/ibis
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing editable metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [36 lines of output]
      Traceback (most recent call last):
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/usr/local/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 181, in prepare_metadata_for_build_editable
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/poetry/core/masonry/api.py", line 42, in prepare_metadata_for_build_wheel
          poetry = Factory().create_poetry(Path(".").resolve(), with_groups=False)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/patch.py", line 26, in alt_poetry_create
          name = _get_and_apply_version(
                 ^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 608, in _get_and_apply_version
          version, instance = _get_version(config, name)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 452, in _get_version
          version = _get_version_from_dunamai(vcs, pattern, config)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/poetry_dynamic_versioning/__init__.py", line 417, in _get_version_from_dunamai
          return Version.from_vcs(
                 ^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1956, in from_vcs
          return cls._do_vcs_callback(
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1995, in _do_vcs_callback
          return callback(**kwargs)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 1916, in from_any_vcs
          vcs = _detect_vcs(None, path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-qu9btf27/overlay/lib/python3.11/site-packages/dunamai/__init__.py", line 375, in _detect_vcs
          raise RuntimeError(dubious_ownership_error)
      RuntimeError: Detected Git repository, but failed because of dubious ownership
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

(this is running from the VS Code terminal window after opening the project as/in a devcontainer)

What version of ibis are you using?

git HEAD 4ad734d

What backend(s) are you using, if any?

No response

Relevant log output

No response

Code of Conduct

I agree to follow this project's Code of Conduct

The text was updated successfully, but these errors were encountered:

ncclementi · 2024-04-18T17:49:20Z

@anentropic Thanks for reporting this.

I'm not quite sure what's going on, because the geospatial dependencies were added recently, see #8837 and it looks like you are running on main.

@cpcloud any idea what could be going on here?

cpcloud · 2024-04-18T17:55:08Z

A local vs code environment a codespace are not the same. If a package attempting to call gdal-config that means it's being built from source, which means the wheel for whatever platform + python version you're using doesn't exist.

cpcloud · 2024-04-18T17:57:17Z

I don't think it's a local versus remote issue.

I also don't really understand how this works. Does the codespace run on GitHub infrastructure and VS code just "points" at that?

cpcloud · 2024-04-18T17:59:23Z

I don't think anyone's used the devcontainer setup locally, but it works on GitHub codespaces. I use it from time to time to try things that I don't want to run locally.

anentropic · 2024-04-18T18:00:20Z

I believe it's running a docker container locally on my machine

from the terminal I get:

vscode ➜ /workspaces/ibis (main) $ arch
aarch64
vscode ➜ /workspaces/ibis (main) $ uname -a
Linux b66611250c8b 6.6.22-linuxkit #1 SMP Fri Mar 29 12:21:27 UTC 2024 aarch64 GNU/Linux

which makes sense as I'm on an M2 macbook

cpcloud · 2024-04-18T18:03:15Z

RuntimeError: Detected Git repository, but failed because of dubious ownership

Not entirely sure whether this will be possible to deal with. Speculating here, but I think this is probably related to how volumes are being mounted inside the container.

cpcloud · 2024-04-18T18:05:40Z

That error is also happening in dunamai which is a dependency of many many things.

I think we probably need someone who develops using a Mac to debug this

anentropic · 2024-04-18T18:24:03Z

for kicks I pasted in the traceback above and asked the new Llama 3 for a solution...

it actually gave a nice explanation and something that worked (!)

amending the update content script to:

POETRY_DYNAMIC_VERSIONING=false python3 -m pip install -e '.[clickhouse,duckdb,clickhouse,examples,geospatial]'

got around the dunamai issue

So the two parts of the fix are probably:

add a Dockerfile under .devcontainer something like:

FROM mcr.microsoft.com/vscode/devcontainers/python:3.11

RUN apt-get update && apt-get install -y --no-install-recommends libgdal-dev

prepend POETRY_DYNAMIC_VERSIONING=false to the pip install -e

…9013) Resolves #9011 take 2 also TIL you can't reopen a closed PR after force-pushing to the branch

gforsyth · 2024-04-18T19:03:15Z

Hey @anentropic -- want to give it another shot now and see if it works?

amoeba · 2024-04-18T22:20:10Z

Hi @gforsyth, I had to make a few tweaks to get the devcontainer to run and those changes are in #9019.

## Description of changes These change are what was required for me to get the devcontainer to run on my machine. Before these changes, the devcontainer image wasn't building correctly because it was missing gdal. The root cause of that was that the devcontainer.json wasn't using the Dockerfile. - The previous devcontainer.json wasn't actually using the Dockerfile, now it does - In devcontainer.json, I switched updateContent command for postCreateCommand as that looks more appropriate based on the docs - A couple of minor tidying changes ## Issues closed Resolves #9011

anentropic · 2024-04-19T10:22:43Z

Hey @anentropic -- want to give it another shot now and see if it works?

Not working yet but looks like #9019 will fix it (needs the change in devcontainer.json to actually use the Dockerfile)

cpcloud · 2024-04-19T10:25:24Z

Just merged that, give it another go if you can!

anentropic · 2024-04-19T16:23:33Z

Apologies, but it seems POETRY_DYNAMIC_VERSIONING=false was a hallucination by the LLM

I tried it and it appeared to work (i.e. by entering that line manually in terminal of a failed container launch)

but it doesn't work when (re)building the container from scratch ...I tried again just now and same thing. (And I can't find evidence for it via Google or GitHub search)

I'm not sure what devcontainers is doing to make the two scenarios behave differently

According to poetry-dynamic-versioning plugin docs a correct env var to disable active versioning would be POETRY_DYNAMIC_VERSIONING_BYPASS='8.0.0' i.e. specifying a static version value

I tried this and took care to make devcontainers rebuild the image with it and it seems to work

cpcloud · 2024-04-20T13:19:39Z

PR to fix the hallucination: #9031

anentropic added the bug Incorrect behavior inside of ibis label Apr 18, 2024

gforsyth mentioned this issue Apr 18, 2024

chore(container): add gdal to codespace #9012

Closed

gforsyth mentioned this issue Apr 18, 2024

chore(container): add gdal to codespace, disable dynamic versioning #9013

Merged

gforsyth closed this as completed in #9013 Apr 18, 2024

gforsyth added a commit that referenced this issue Apr 18, 2024

chore(container): add gdal to codespace, disable dynamic versioning (#…

08a33e9

…9013) Resolves #9011 take 2 also TIL you can't reopen a closed PR after force-pushing to the branch

amoeba mentioned this issue Apr 18, 2024

fix: make devcontainer work correctly #9019

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: devcontainer is missing GDAL #9011

bug: devcontainer is missing GDAL #9011

anentropic commented Apr 18, 2024

ncclementi commented Apr 18, 2024

cpcloud commented Apr 18, 2024

cpcloud commented Apr 18, 2024 •

edited

Loading

cpcloud commented Apr 18, 2024

anentropic commented Apr 18, 2024

cpcloud commented Apr 18, 2024

cpcloud commented Apr 18, 2024

anentropic commented Apr 18, 2024 •

edited

Loading

gforsyth commented Apr 18, 2024

amoeba commented Apr 18, 2024

anentropic commented Apr 19, 2024

cpcloud commented Apr 19, 2024

anentropic commented Apr 19, 2024

cpcloud commented Apr 20, 2024

bug: devcontainer is missing GDAL #9011

bug: devcontainer is missing GDAL #9011

Comments

anentropic commented Apr 18, 2024

What happened?

What version of ibis are you using?

What backend(s) are you using, if any?

Relevant log output

Code of Conduct

ncclementi commented Apr 18, 2024

cpcloud commented Apr 18, 2024

cpcloud commented Apr 18, 2024 • edited Loading

cpcloud commented Apr 18, 2024

anentropic commented Apr 18, 2024

cpcloud commented Apr 18, 2024

cpcloud commented Apr 18, 2024

anentropic commented Apr 18, 2024 • edited Loading

gforsyth commented Apr 18, 2024

amoeba commented Apr 18, 2024

anentropic commented Apr 19, 2024

cpcloud commented Apr 19, 2024

anentropic commented Apr 19, 2024

cpcloud commented Apr 20, 2024

cpcloud commented Apr 18, 2024 •

edited

Loading

anentropic commented Apr 18, 2024 •

edited

Loading