Make the channel importer versions dynamic #455

dasJ · 2022-03-29T17:54:25Z

I have no idea if this works…

Taken from: https://thekevinwang.com/2021/09/19/github-actions-dynamic-matrix/

ysndr · 2022-04-02T12:51:57Z

Oh ive been looking for this for a while.
I thought you couldn't use expressions at the matrix side.

This way it will be easier to update them because all the information is in one JSON blob (inside Channels.elm file). Ref: NixOS#455

garbas · 2022-04-07T15:23:22Z

The channels are actually coming from this channels.nix file. I would prefer if we use that way of getting the channels.

Here would be the full plan:

1. add nixos-org-configurations as input to flake.nix
2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON.
3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION.
4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there.
5. we should then add a nixosChannels to Flags type alias in frontend/src/Main.elm and store it in the mode in the init function in the same file. We should also decode the JSON that is being passes in to ensure it is of correct shape.
6. we should then change how we handle Channel in frontend/src/Search.elm to not type check anymore but to get all the information from the model.nixosChannels.
7. also use NIXOS_CHANNELS variable with flake-info

Would somebody be willing to work on this? I can help out if you get stuck, either via matrix or short call.

/cc @dasJ @karantan

1. add nixos-org-configurations as input to flake.nix 2. evaluate channels.nix file and export channels via environment variable. that environment variable (lets call it NIXOS_CHANNELS) should be present during the build and inside the nix shell. the content of the variable can be JSON. 3. we pickup the NIXOS_CHANNELS environment variable in frontend/webpack.config.js and pass it further to webpack process, just like we do with ELASTICSEARCH_MAPPING_SCHEMA_VERSION. 4. we forward NIXOS_CHANNELS to Elm via frontend/src/index.js as an Elm application flag. Just like we do with other variables there.

garbas · 2022-04-07T22:42:09Z

I went ahead and merge with current main and got first 4 items done.

ncfavier · 2022-04-08T08:41:28Z

I think this would mean we only get new channels when we update the flake inputs? Even if that process is semi-automated, I wonder if we could update the channels truly dynamically.

garbas · 2022-04-08T09:27:43Z

I think this would mean we only get new channels when we update the flake inputs? Even if that process is semi-automated, I wonder if we could update the channels truly dynamically.

I would like to have control over the channels. To help us reproduce the bugs. I don't think it is a problem if channel comes online a day later.

flake.nix

karantan · 2022-04-09T09:34:37Z

Would somebody be willing to work on this? I can help out if you get stuck, either via matrix or short call.

🤚 I think I can do it. Or at least give it a shot. I would also like to go on a short call if it's possible to make sure I'm going in the right direction.

garbas · 2022-04-10T20:28:42Z

Would somebody be willing to work on this? I can help out if you get stuck, either via matrix or short call.

raised_back_of_hand I think I can do it. Or at least give it a shot. I would also like to go on a short call if it's possible to make sure I'm going in the right direction.

Join us here and lets see when we both have time: https://matrix.to/#/#nixos-search:matrix.org

flake-info/src/bin/flake-info.rs

ysndr · 2022-04-17T11:47:56Z

Pushed to the wrong repo, point 7 now in here too

ysndr · 2022-04-17T14:28:35Z

I split the dev shells for frontend and flake info in the last commit since i dont think you should need to down load 4G of elm if you only work on the rust and vice verse.
Happy to make a new PR on top of this later if that is desired.

.github/workflows/import-to-elasticsearch.yml

flake.nix

ncfavier · 2022-04-17T15:08:06Z

flake-info/src/bin/flake-info.rs

+/// Information about allowed and default nixos channels.
+/// Typyically passed by environment variable NIXOS_CHANNELS.
+/// Used to filter the input arguments for `flake-info nixpkgs` and `flake-info nixpkgs-archive`


Not sure I understand the point of checking this in flake-info. What if I want to run nixpkgs-archive locally on some random branch? Will that fail?

Co-authored-by: Naïm Favier <n@monade.li>

but still provide devShells for each of the subprojects

and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION;

Co-authored-by: Naïm Favier <n@monade.li>

garbas · 2022-04-20T19:15:33Z

Instead of writing a JSON file, how about instead exposing the nixos channels as a lib.nixosChannels output (defined outside of the eachDefaultSystems call), and in the workflow doing something like

I plan to reuse this JSON file also for other flakes. I guess we should rename it than to something else, but I guess we can do this at the later point in time, but this way we are prepared to grow the channels (or release branches) for other things then nixpkgs.

garbas · 2022-04-20T19:18:33Z

I split the dev shells for frontend and flake info in the last commit since i dont think you should need to down load 4G of elm if you only work on the rust and vice verse.
Happy to make a new PR on top of this later if that is desired.

I've reverted partially the change and made that both subprojects are there in devShell.default, but you can always select a subproject (nix develop .#flake-info) if you don't with to pull in the Elm stuff.

…to feat/dynamic-import

garbas · 2022-04-14T07:49:03Z

flake-info/src/bin/flake-info.rs

@@ -237,6 +237,9 @@ async fn run_command(
            Ok((exports, ident))
        }
        Command::Nixpkgs { channel } => {
+            // TODO: if NIXOS_CHANNELS environment variable is present check


@karantan @ysndr This is where step number 7 should be implemented. I've also added the instructions

garbas · 2022-04-20T19:40:04Z

I'm closing this PR and open new one to provide a preview for changes.

Make the channel importer versions dynamic

39d768c

dasJ force-pushed the feat/dynamic-import branch from 7d792a3 to 39d768c Compare March 29, 2022 17:56

karantan added a commit to karantan/nixos-search that referenced this pull request Apr 7, 2022

Move hardcoded nix channels to Channels.elm

dba5397

This way it will be easier to update them because all the information is in one JSON blob (inside Channels.elm file). Ref: NixOS#455

karantan added a commit to karantan/nixos-search that referenced this pull request Apr 7, 2022

Move hardcoded nix channels to Channels.elm

bac2385

This way it will be easier to update them because all the information is in one JSON blob (inside Channels.elm file). Ref: NixOS#455

karantan mentioned this pull request Apr 7, 2022

Move hardcoded nix channels to Channels.elm #465

Closed

karantan added a commit to karantan/nixos-search that referenced this pull request Apr 7, 2022

Move hardcoded nix channels to Channels.elm

ff2e1ae

This way it will be easier to update them because all the information is in one JSON blob (inside Channels.elm file). Ref: NixOS#455

karantan added a commit to karantan/nixos-search that referenced this pull request Apr 7, 2022

Move hardcoded nix channels to Channels.elm

99f6a95

This way it will be easier to update them because all the information is in one JSON blob (inside Channels.elm file). Ref: NixOS#455

garbas self-requested a review April 7, 2022 15:23

This was referenced Apr 7, 2022

Add support for beta channels #453

Closed

Add manual channel updater #456

Closed

garbas added 2 commits April 7, 2022 23:59

Merge remote-tracking branch 'origin/main' into feat/dynamic-import

e48ea62

Decode nixosChannels in Elm

e3d3fc1

ncfavier self-requested a review April 8, 2022 09:29

ncfavier reviewed Apr 8, 2022

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

garbas added 5 commits April 12, 2022 22:12

Use nixosChannels that came via application flag

de30398

Merge remote-tracking branch 'origin/main' into feat/dynamic-import

72748d8

read nixos channels in github action

28a2d4d

defaultNixOSChannel should be calculated

6a50ea0

add two pointers where the check should be added

b45ccc0

garbas reviewed Apr 14, 2022

View reviewed changes

flake-info/src/bin/flake-info.rs Outdated Show resolved Hide resolved

garbas and others added 2 commits April 15, 2022 23:24

pass nixosChannels to flake-info and remove title, rather calculate it

5ce774b

Add NixosChannels struct validation and validation Error

898cf31

ysndr added 3 commits April 17, 2022 13:44

Read NIXOS_CHANNEL variable

b23fc2d

Check channel

a631802

Add channel struct to fix parsing NIXOS_CHANNELS

f14ff5f

ysndr added 2 commits April 17, 2022 16:26

Use eachDefaultSystem instead of listing them manually

ecce987

Add individual dev shells for frontend and backend

638da42

ncfavier requested changes Apr 17, 2022

View reviewed changes

garbas and others added 5 commits April 20, 2022 20:54

Update .github/workflows/import-to-elasticsearch.yml

893d01c

Co-authored-by: Naïm Favier <n@monade.li>

use both development environments by default (as it was)

687cc09

but still provide devShells for each of the subprojects

pkgs.lib → lib everywhere

ebb94a3

and define lib = nixpkgs.lib before the call to eachDefaultSystem Also, version = lib.fileContents ./VERSION;

Update flake.nix

a7645ae

Co-authored-by: Naïm Favier <n@monade.li>

Merge remote-tracking branch 'origin/main' into feat/dynamic-import

50c4eb7

garbas added 2 commits April 20, 2022 21:18

Merge branch 'feat/dynamic-import' of github.com:dasJ/nixos-search in…

9fd5827

…to feat/dynamic-import

typo

82bbcd3

garbas approved these changes Apr 20, 2022

View reviewed changes

garbas closed this Apr 20, 2022

garbas mentioned this pull request Apr 20, 2022

Feat/dynamic import #473

Merged

github-actions bot temporarily deployed to pull request April 20, 2022 19:42 Inactive

ncfavier mentioned this pull request Dec 23, 2022

flake-info: remove broken NIXOS_CHANNELS check #586

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the channel importer versions dynamic #455

Make the channel importer versions dynamic #455

dasJ commented Mar 29, 2022

ysndr commented Apr 2, 2022

garbas commented Apr 7, 2022 •

edited

garbas commented Apr 7, 2022

ncfavier commented Apr 8, 2022

garbas commented Apr 8, 2022

karantan commented Apr 9, 2022

garbas commented Apr 10, 2022

ysndr commented Apr 17, 2022

ysndr commented Apr 17, 2022

ncfavier Apr 17, 2022

garbas commented Apr 20, 2022

garbas commented Apr 20, 2022

garbas Apr 14, 2022

garbas commented Apr 20, 2022

Make the channel importer versions dynamic #455

Make the channel importer versions dynamic #455

Conversation

dasJ commented Mar 29, 2022

ysndr commented Apr 2, 2022

garbas commented Apr 7, 2022 • edited

garbas commented Apr 7, 2022

ncfavier commented Apr 8, 2022

garbas commented Apr 8, 2022

karantan commented Apr 9, 2022

garbas commented Apr 10, 2022

ysndr commented Apr 17, 2022

ysndr commented Apr 17, 2022

ncfavier Apr 17, 2022

Choose a reason for hiding this comment

garbas commented Apr 20, 2022

garbas commented Apr 20, 2022

garbas Apr 14, 2022

Choose a reason for hiding this comment

garbas commented Apr 20, 2022

garbas commented Apr 7, 2022 •

edited