spider: init at 2.33.11 #389848

KSJ2000 · 2025-03-14T14:50:04Z

Resolves #388120

Things done

Add a 👍 reaction to pull requests you find important.

as per spider-rs/spider#276 (comment)

ethancedwards8

LGTM. Thanks.

pkgs/by-name/sp/spider/package.nix

GaetanLepage · 2025-03-16T11:16:00Z

pkgs/by-name/sp/spider/package.nix

+  fetchFromGitHub,
+  lib,
+  nix-update-script,
+  openssl,
+  pkg-config,
+  rust-jemalloc-sys,
+  rustPlatform,
+  stdenv,
+  sqlite,
+  versionCheckHook,
+  zstd,


Please, sort the inputs in the order of apparition in the derivation (except lib which should remain at the top).

Is this written somewhere? because I think alphabetical order is easy to read/find and to add new as needed, so pretty logical.
Probably only chinese people would not agree.

No, I don't think it is written in an official document.
It is just significantly more consensual within the code base.
Now, if you prefer having it this way, I will not block this PR for this reason.

GaetanLepage · 2025-03-16T11:16:09Z

pkgs/by-name/sp/spider/package.nix

+  pkg-config,
+  rust-jemalloc-sys,
+  rustPlatform,
+  stdenv,


Suggested change

stdenv,

unused.

most of the package is taken from nix-init

nix-init does not put lib at the top?

the alphabetical order around the package was my doing when I removed some deprecated darwin features and added a some for checks
nix-init added stdenv which I did not see it's not used
definitely needs an update

Ok understood. nix-init would need to be updated indeed.
The "apparition ordering" is way more common across the nixpkgs code base. This is why I asked you to adopt it.

GaetanLepage · 2025-03-16T14:42:16Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 389848

`x86_64-linux`

✅ 1 package built:

spider

`aarch64-linux`

✅ 1 package built:

spider

`x86_64-darwin`

❌ 1 package failed to build:

spider

`aarch64-darwin`

❌ 1 package failed to build:

spider

KSJ2000 · 2025-03-17T07:46:08Z

That is quite weird to fail on darwin because I built it fine on aarch64-darwin.

`nixpkgs-review` result

Generated using nixpkgs-review-gha

Command: nixpkgs-review pr 389848

Logs: https://github.com/KSJ2000/nixpkgs-review-gha/actions/runs/13893515914

`x86_64-linux`

✅ 1 package built:

spider

`aarch64-linux`

✅ 1 package built:

spider

`x86_64-darwin`

✅ 1 package built:

spider

`aarch64-darwin`

✅ 1 package built:

spider

GaetanLepage · 2025-03-18T13:04:38Z

I keep getting those failures on the darwin community builders:

test website::test_link_duplicates ... FAILED
test website::test_crawl_subscription ... FAILED
test website::test_crawl_budget ... FAILED
test website::not_crawl_blacklist ... FAILED

failures:

---- website::test_link_duplicates stdout ----

thread 'website::test_link_duplicates' panicked at /private/tmp/nix-build-spider-2.33.11.drv-0/spider-2.33.11-vendor/system-configuration-0.6.1/src/dynamic_store.rs:154:1:
Attempted to create a NULL object.

---- website::test_crawl_subscription stdout ----

thread 'website::test_crawl_subscription' panicked at /private/tmp/nix-build-spider-2.33.11.drv-0/spider-2.33.11-vendor/system-configuration-0.6.1/src/dynamic_store.rs:154:1:
Attempted to create a NULL object.
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- website::test_crawl_budget stdout ----

thread 'website::test_crawl_budget' panicked at /private/tmp/nix-build-spider-2.33.11.drv-0/spider-2.33.11-vendor/system-configuration-0.6.1/src/dynamic_store.rs:154:1:
Attempted to create a NULL object.

---- website::not_crawl_blacklist stdout ----

thread 'website::not_crawl_blacklist' panicked at /private/tmp/nix-build-spider-2.33.11.drv-0/spider-2.33.11-vendor/system-configuration-0.6.1/src/dynamic_store.rs:154:1:
Attempted to create a NULL object.


failures:
    website::not_crawl_blacklist
    website::test_crawl_budget
    website::test_crawl_subscription
    website::test_link_duplicates

test result: FAILED. 21 passed; 4 failed; 0 ignored; 0 measured; 8 filtered out; finished in 0.01s

I tried to add __darwinAllowLocalNetworking = true; but it doesn't seem to help. Are you building it with the sandbox disabled?

KSJ2000 · 2025-03-18T14:51:25Z

That seems to be the case.
I haven't changed anything since installing nix, but nix-info -m shows sandbox: no

I tried enabling it with the following:

export NIX_SANDBOX=1
add sandbox = true to /etc/nix/nix.conf

Neither have changed sandbox to true
Last thing I found was adding --option sandbox true to nix-build, but it gives an error:

warning: Ignoring the client-specified setting 'sandbox', because it is a restricted setting and you are not a trusted user

I'm guessing I should add those failing tests to checkFlags list.
Is there a way to enable sandbox on darwin? I don't understand why it's not enabled by default.

EDIT: Need to reboot system for changes to /etc/nix/nix.conf to take effect!

ethancedwards8 · 2025-03-18T14:54:20Z

Add your user to trusted-users in nix.conf. For example, mine is:

trusted-users = root ece @wheel @admin

The @ means a group.

GaetanLepage · 2025-03-18T15:02:38Z

pkgs/by-name/sp/spider/package.nix

+  # Network and io_uring not available in sandbox
+  checkFlags = [
+    "--skip=website::crawl"


Suggested change

# Network and io_uring not available in sandbox

checkFlags = [

"--skip=website::crawl"

checkFlags = [

# Network and io_uring not available in sandbox

"--skip=website::crawl"

nit

GaetanLepage · 2025-03-18T15:05:50Z

pkgs/by-name/sp/spider/package.nix

+    "--skip=page::test_status_code"
+    "--skip=pdl_is_fresh"
+    "--skip=verify_revision_available"
+  ];


Suggested change

];

] ++ lib.optionals stdenv.isDarwin [

# Fail in the darwin sandbox with:

# Attempted to create a NULL object.

"--skip=website::not_crawl_blacklist"

"--skip=website::test_crawl_budget"

"--skip=website::test_crawl_subscription"

"--skip=website::test_link_duplicates"

];

Why did you add more tests than the one listed in my comment? Were they failing too?
Also, I think that it is relevant to mention that it is a sandbox limitation. You have only written the error message.

Yes, they were failing. I added everything until the package built.
I thought it was obvious those were fails in the darwin sandbox because of the ++ lib.optionals stdenv.isDarwin
Sandbox limitation: Attempted to create a NULL object
would probably be a better message, right? just like io_uring which is available outside the sandbox

Sandbox limitation: Attempted to create a NULL object

This would be clearer indeed.

KSJ2000 · 2025-03-18T15:21:31Z

It is not enough to add trusted-users to /etc/nix/nix.conf which is what I also tried. You need to reboot the system too, for it to take effect.

The weird thing is that even ofborg doesn't have the sandbox enabled since it built the package just fine:
https://logs.ofborg.org/?attempt_id=e8be082e-62ad-4f9d-967e-685b7981679f&key=nixos%2Fnixpkgs.389848
https://logs.ofborg.org/?attempt_id=0713a389-e2b8-443b-9467-3360a302d548&key=nixos%2Fnixpkgs.389848

GaetanLepage · 2025-03-18T15:36:03Z

The weird thing is that even ofborg doesn't have the sandbox enabled since it built the package just fine:

No, ofborg does not enable the darwin sandbow indeed. So many things would be broken otherwise...

GaetanLepage · 2025-03-18T16:45:57Z

`nixpkgs-review` result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 389848

`x86_64-linux`

✅ 1 package built:

spider

`aarch64-linux`

✅ 1 package built:

spider

`x86_64-darwin`

✅ 1 package built:

spider

`aarch64-darwin`

✅ 1 package built:

spider

GaetanLepage

LGTM, thanks

maintainers: add j-mendez

8330994

as per spider-rs/spider#276 (comment)

github-actions bot added 8.has: maintainer-list (update) 10.rebuild-darwin: 1 10.rebuild-darwin: 1-10 10.rebuild-linux: 1 10.rebuild-linux: 1-10 labels Mar 14, 2025

ethancedwards8 approved these changes Mar 15, 2025

View reviewed changes

KSJ2000 mentioned this pull request Mar 16, 2025

spider: init at 2.33.11 #389981

Closed

13 tasks

GaetanLepage reviewed Mar 16, 2025

View reviewed changes

pkgs/by-name/sp/spider/package.nix Outdated Show resolved Hide resolved

KSJ2000 force-pushed the spider branch from 6d6d678 to 339e129 Compare March 16, 2025 11:13

GaetanLepage reviewed Mar 16, 2025

View reviewed changes

KSJ2000 force-pushed the spider branch from 339e129 to 85bc68f Compare March 16, 2025 11:30

GaetanLepage reviewed Mar 18, 2025

View reviewed changes

KSJ2000 force-pushed the spider branch from 85bc68f to 48ac5de Compare March 18, 2025 15:19

KSJ2000 mentioned this pull request Mar 18, 2025

Darwin builders should enable nix sandbox Defelo/nixpkgs-review-gha#4

Closed

spider: init at 2.33.11

86e400b

KSJ2000 force-pushed the spider branch from 48ac5de to 86e400b Compare March 18, 2025 15:52

GaetanLepage approved these changes Mar 18, 2025

View reviewed changes

GaetanLepage merged commit be1654a into NixOS:master Mar 18, 2025
27 of 30 checks passed

KSJ2000 deleted the spider branch March 19, 2025 07:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

spider: init at 2.33.11 #389848

spider: init at 2.33.11 #389848

KSJ2000 commented Mar 14, 2025

ethancedwards8 left a comment

GaetanLepage Mar 16, 2025

KSJ2000 Mar 16, 2025

GaetanLepage Mar 16, 2025

GaetanLepage Mar 16, 2025

KSJ2000 Mar 16, 2025

GaetanLepage Mar 16, 2025

KSJ2000 Mar 16, 2025

GaetanLepage Mar 16, 2025

GaetanLepage commented Mar 16, 2025

KSJ2000 commented Mar 17, 2025

GaetanLepage commented Mar 18, 2025

KSJ2000 commented Mar 18, 2025 •

edited

Loading

ethancedwards8 commented Mar 18, 2025

GaetanLepage Mar 18, 2025

GaetanLepage Mar 18, 2025

GaetanLepage Mar 18, 2025

KSJ2000 Mar 18, 2025

GaetanLepage Mar 18, 2025

KSJ2000 commented Mar 18, 2025 •

edited

Loading

GaetanLepage commented Mar 18, 2025

GaetanLepage commented Mar 18, 2025

GaetanLepage left a comment

-  ];
+  ] ++ lib.optionals stdenv.isDarwin [
+    # Fail in the darwin sandbox with:
+    # Attempted to create a NULL object.
+    "--skip=website::not_crawl_blacklist"
+    "--skip=website::test_crawl_budget"
+    "--skip=website::test_crawl_subscription"
+    "--skip=website::test_link_duplicates"
+  ];

spider: init at 2.33.11 #389848

spider: init at 2.33.11 #389848

Conversation

KSJ2000 commented Mar 14, 2025

Things done

ethancedwards8 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GaetanLepage commented Mar 16, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

KSJ2000 commented Mar 17, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

GaetanLepage commented Mar 18, 2025

KSJ2000 commented Mar 18, 2025 • edited Loading

ethancedwards8 commented Mar 18, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KSJ2000 commented Mar 18, 2025 • edited Loading

GaetanLepage commented Mar 18, 2025

GaetanLepage commented Mar 18, 2025

nixpkgs-review result

x86_64-linux

aarch64-linux

x86_64-darwin

aarch64-darwin

GaetanLepage left a comment

Choose a reason for hiding this comment

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`

KSJ2000 commented Mar 18, 2025 •

edited

Loading

KSJ2000 commented Mar 18, 2025 •

edited

Loading

`nixpkgs-review` result

`x86_64-linux`

`aarch64-linux`

`x86_64-darwin`

`aarch64-darwin`