Skip to content

Comments

Prefect init module#386895

Merged
happysalada merged 2 commits intoNixOS:masterfrom
happysalada:prefect_init_module
Mar 19, 2025
Merged

Prefect init module#386895
happysalada merged 2 commits intoNixOS:masterfrom
happysalada:prefect_init_module

Conversation

@happysalada
Copy link
Contributor

Things done

I'm close with the module but I still have errors on the worker.
I'm opening this in case someone has some cycles to help.
The prefect server is working but the worker is crashlooping.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@happysalada happysalada marked this pull request as draft March 4, 2025 01:56
@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (update) This PR changes an existing module in `nixos/` labels Mar 4, 2025
@happysalada
Copy link
Contributor Author

I submitted the prefect patch upstream and it was accepted, so the patch is just temporary.

@github-actions github-actions bot added 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Mar 4, 2025
@happysalada
Copy link
Contributor Author

@nviets just in case you are interested in this.
I'm still debugging the worker service, but the server is working, if you notice anything I've missed, let me know.

@h7x4 h7x4 added the 8.has: module (new) This PR adds a module in `nixos/` label Mar 11, 2025
@Shihira
Copy link

Shihira commented Mar 14, 2025

@happysalada Thank you for your package and module, they are very helpful. I succeeded to get the worker service to work as well, and the crash I observed was mainly because the worker spawned another python process that didn't inherit any python package from the main prefect process. My approach was to wrap the prefect environment with a devShell, but maybe the better way could be to use makeWrapper with some fine-tuned arguments.

@happysalada
Copy link
Contributor Author

I dont get why its trying to spin up an ephemeral server though, if i give it the api url, it should just call that. Im going to wrap the python for the missing uvicorn package this weekend to see what happens.
I suspect this wont fix the problem though as it looka like the workers wont be able to find the main server. Ill try to update this this weekend

@happysalada happysalada force-pushed the prefect_init_module branch from dd7d571 to 8bc7fbb Compare March 15, 2025 12:14
prefect: add wrapper for dependencies
@happysalada happysalada force-pushed the prefect_init_module branch from 8bc7fbb to f0fd7d7 Compare March 15, 2025 12:31
@happysalada
Copy link
Contributor Author

@Shihira ok, I got to the next step, I'm getting a database error "could not open database". i guess I have to add the sqlite path to the readwrite of the worker maybe.
I'm not sure I'll have more time this weekend, but I'll look at this next.

@happysalada happysalada force-pushed the prefect_init_module branch 10 times, most recently from 908d9c1 to 2baeb00 Compare March 16, 2025 15:03
prefect: add dburl to worker

prefect: use same state directory

prefect: fix worker environment

prefect: create user

prefect: use datadir for sqlite url

prefect: make datadir writable

prefect: don't protect home

prefect fix sqlite url

prefect: fix state directory

prefect: user should not be systemuser

prefect: set to normal user

add prefect to systempackages

try user with same name

prefect use prefect_home

do not set database url

revert to dynamic user

prefect: add tests

prefect: fix port to string
@happysalada happysalada force-pushed the prefect_init_module branch from 2baeb00 to ef12e14 Compare March 16, 2025 15:09
@happysalada happysalada marked this pull request as ready for review March 16, 2025 15:17
@happysalada
Copy link
Contributor Author

Ok, I was getting errors when setting the database url environment variable.
For now I'm leaving it not setup to use the default.
I'm going to use this some more before potentially finding a fix.
I think this is ready for a merge, with some potential improvements in database setup left for future updates.

@happysalada happysalada merged commit a72f22d into NixOS:master Mar 19, 2025
39 checks passed
@happysalada happysalada deleted the prefect_init_module branch March 19, 2025 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: module (new) This PR adds a module in `nixos/` 8.has: module (update) This PR changes an existing module in `nixos/` 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-darwin: 1 This PR causes 1 package to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants