support http_proxy (was: Pages does not load, segfault on exit)

[eaubin]

Trying duckdb-ui for the first time with v.1.2.1 8e52ec4395 extension version 963e0e4 on osx arm. The extension loads and gives the 'UI started at http://localhost:4213/' but the server doesn't respond. I've tried both chrome and curl.

Also, exiting the duckdb cli with Ctrl-C multiple times gives a segfault.

[Y--]

Hi @eaubin! Thanks for reporting the issue and sorry you're facing this.

It looks like you might have an outdated, cached version.

Could you possibly try to remove the ui extension (with rm ~/.duckdb/extensions/v1.2.1/osx_arm64/ui.duckdb_extension*) and try again?

Could you also send the exact steps you took to start the UI? Did you do duckdb -ui or did you CALL start_ui_server() from the CLI?

Finally, if you could send the output of curl -vvv http://localhost:4213 and a screenshot of your Chrome console if it shows any error, it would be super helpful. Thanks!

[eaubin]

Here's the full trace

❯ rm .duckdb/extensions/v1.2.1/osx_arm64/ui*
❯ brew reinstall duckdb
==> Downloading https://ghcr.io/v2/homebrew/core/duckdb/manifests/1.2.1
Already downloaded: $HOME/Library/Caches/Homebrew/downloads/c23f4c6a9c97daa0a82b22480bfda02a667825163cd496ca726d041fde3b8ec2--duckdb-1.2.1.bottle_manifest.json
==> Fetching duckdb
==> Downloading https://ghcr.io/v2/homebrew/core/duckdb/blobs/sha256:ec3a38cd6619f40094000b8909d750aefce1c7febe0ba36930afdb06
Already downloaded: $HOME/Library/Caches/Homebrew/downloads/126d3f7672d19759def5f13a1e0cf2b74bfc5b36879c7e6e88d24ddcc1c0620b--duckdb--1.2.1.arm64_sequoia.bottle.tar.gz
==> Reinstalling duckdb
==> Pouring duckdb--1.2.1.arm64_sequoia.bottle.tar.gz
🍺  /opt/homebrew/Cellar/duckdb/1.2.1: 1,217 files, 158.3MB
==> Running `brew cleanup duckdb`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
❯ duckdb
v1.2.1 8e52ec4395
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D SET http_proxy = 'MY_PROXY_HERE';
D CALL start_ui();
┌──────────────────────────────────────┐
│                result                │
│               varchar                │
├──────────────────────────────────────┤
│ UI started at http://localhost:4213/ │
└──────────────────────────────────────┘
D select * FROM duckdb_extensions() WHERE extension_name='ui';
┌────────────────┬─────────┬───────────┬───┬───────────┬───────────────────┬──────────────┬────────────────┐
│ extension_name │ loaded  │ installed │ … │  aliases  │ extension_version │ install_mode │ installed_from │                 │    varchar     │ boolean │  boolean  │   │ varchar[] │      varchar      │   varchar    │    varchar     │
├────────────────┼─────────┼───────────┼───┼───────────┼───────────────────┼──────────────┼────────────────┤
│ ui             │ true    │ true      │ … │ []        │ 963e0e4           │ REPOSITORY   │ core           │
├────────────────┴─────────┴───────────┴───┴───────────┴───────────────────┴──────────────┴────────────────┤
│ 1 rows                                                                               9 columns (7 shown) │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────┘
# From a new terminal
❯ curl -vvv http://localhost:4213
* Uses proxy env variable no_proxy == '127.0.0.1,localhost,$MYPROXY'
* Host localhost:4213 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:4213...
* connect to ::1 port 4213 from ::1 port 57290 failed: Connection refused
*   Trying 127.0.0.1:4213...
* Connected to localhost (127.0.0.1) port 4213
> GET / HTTP/1.1
> Host: localhost:4213
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off

The Chrome developer log shows one pending request that doesn't get answered. What version of the extension where you expecting? I could try downloading and installing it manually. duckdb -ui behaves the same as CALL start_ui();

[eaubin]

Looking at the docs, it says that it fetches files from a remote HTTP server to be kept up to date, does the extension respect the duckdb http_proxy variable?

[jraymakers]

Looking at the docs, it says that it fetches files from a remote HTTP server to be kept up to date, does the extension respect the duckdb http_proxy variable?

Hm, no, I don't think it does. If you're depending on that, it could be the issue. Could you say more about your use of http_proxy?

[eaubin]

At $work all network access is through a proxy, I can't access the internet directly. Does the ui cache the downloaded content? I could manually download it to the right place on disk.

[jraymakers]

At $work all network access is through a proxy, I can't access the internet directly. Does the ui cache the downloaded content? I could manually download it to the right place on disk.

I see. The content isn't explicitly downloaded, though, once fetched, the browser should cache most (but not all) of it.

Probably a change would be needed to the extension to properly support proxies.

[eaubin]

fwiw, making it work offline would be nice for those times and environments where there is no internet access.

[jraymakers]

fwiw, making it work offline would be nice for those times and environments where there is no internet access.

Agreed. This is on our radar, but we'd have to figure out a few details first.

[curio77]

Having the same issue. I suspect the CLI/backend does HTTP requests of its own that don't respect the http_proxy setting.

[jraymakers]

Having the same issue. I suspect the CLI/backend does HTTP requests of its own that don't respect the http_proxy setting.

Yes, this is correct. We'll need to investigate how to properly respect this setting.

[mlafeldt]

I shared a short guide on how to use the DuckDB UI without internet access here.

Speaking of proxies, this straightforward method is based on traffic capture/replay via the excellent mitmproxy.

[pbrady32]

+1 to this issue. Our HPC compute nodes are behind an http proxy as well. Would love to be able to use the UI on our system.