`lib.fileset.intersection`: init #257356

infinisil · 2023-09-26T00:21:37Z

Description of changes

This is another split off from the WIP fileset combinators PR.
This adds one function:

lib.fileset.intersection: Computes the intersection between two file sets

This work is sponsored by Antithesis ✨

Things done

Tests
Documentation

infinisil · 2023-10-03T22:49:07Z

lib/fileset/README.md

+### No intersection for lists
+
+While there is `intersection a b`, there is no function `intersections [ a b c ]`.
+
+Arguments:
+- (+) There is no known use case for such a function, it can be added later if a use case arises
+- (+) There is no suitable return value for `intersections [ ]`, see also "Nullary intersections" [here](https://en.wikipedia.org/w/index.php?title=List_of_set_identities_and_relations&oldid=1177174035#Definitions)
+  - (-) Could throw an error for that case
+  - (-) Create a special value to represent "all the files" and return that
+    - (+) Such a value could then not be used with `fileFilter` unless the internal representation is changed considerably
+  - (-) Could return the empty file set
+    - (+) This would be wrong in set theory
+- (-) Inconsistent with `union` and `unions`


@roberth @fricklerhandwerk This PR isn't done yet, but I wonder what you think of this design decision.

It's an interesting duality:

Union is mostly needed between many file sets (e.g. combining together the exact files you need)

Intersection is mostly needed between few file sets (e.g. limiting the selected files to some directory)

We might really only need unions (taking a list) and intersection (binary operation).
union (binary operation) doesn't seem that important, but I can imagine some situations where it could be useful and we have it already.

Notably Haskell's Data.Set also provides union, unions, intersection but no intersections (there is one, but it's in Data.Set.Internal).

We could have an interface that requires at least one set: intersect :: a -> [a] -> a. But probably a binary one is good enough. One can still fold over a list, then defining the base set will be required for sane results. That again we can add to documentation as an example.

In math it's easy to invoke a somewhat implicitly defined "universe" set. Us finite mortals would have to make that explicit, which is another way to land on a -> List a -> a whereas @fricklerhandwerk seems to have arrived through NonEmptyList a -> a.

infinisil · 2023-10-04T19:37:21Z

This is ready to review now. There's tests and documentation now.

infinisil · 2023-10-10T16:15:27Z

@roberth This PR is ready to be merged imo, so I'm planning to merge it myself in a couple days unless you have some concerns until then. And if it looks good to you I wouldn't mind it being merged earlier of course :)

roberth · 2023-10-11T10:34:12Z

lib/fileset/README.md

+
+Arguments:
+- Alternative: Use the common prefix of all base paths as the resulting base path
+  - (-) This is unnecessarily strict, because the purpose of the base path is to track the directory under which files _could_ be in the file set.


Wasn't it already defined to be the most specific, longest path that contains them all? Or is it not feasible to make that truly an invariant?

It's not defined like that yet. It's a bit tricky because file sets can be empty but still have a base path, but I think it could be done with some careful wording

lib/fileset/README.md

roberth

A suggestion and some thoughts. LGTM.

roberth · 2023-10-11T10:49:37Z

lib/fileset/default.nix

+
+    Example:
+      # Limit the selected files to the ones in ./., so only ./src and ./Makefile
+      intersection ./. (unions [ ../LICENSE ./src ./Makefile ])


For a more realistic example that readily applies in many projects:

let buildFiles = fileFilter (f: f.ext != "nix") ../.; in intersection buildFiles (unions [ ../build-support ./src ./Makefile ])

Not sure if I got the filter right. Had to change LICENSE in order to realistically encounter .nix files in the union.

Initially I wrote the fileFilter without a path (../.). This would have been feasible if in a design where we allow infinite, non-enumerable sets such as intersections [ ].

buildFiles is particularly useful if you have multiple derivations that each build some part of that set.

fileFilter doesn't exist yet, but I just opened the PR for it yesterday: #260265

I can add this example in the PR once this one is merged

Co-authored-by: Robert Hensing <robert@roberthensing.nl>

While this change is backwards-incompatible, I think it's okay because: - The `fileFilter` function is not yet in a stable NixOS release, it was only merged about [a month ago](NixOS#257356). - All public uses of the function on GitHub only pass a path - Any `fileFilter pred fileset` can also be expressed as `intersection fileset (fileFilter pred path)` without loss of functionality. - This is furthermore pointed out in the new error message when a file set is passed

While this change is backwards-incompatible, I think it's okay because: - The `fileFilter` function is not yet in a stable NixOS release, it was only merged about [a month ago](NixOS/nixpkgs#257356). - All public uses of the function on GitHub only pass a path - Any `fileFilter pred fileset` can also be expressed as `intersection fileset (fileFilter pred path)` without loss of functionality. - This is furthermore pointed out in the new error message when a file set is passed

github-actions bot added the 6.topic: lib The Nixpkgs function library label Sep 26, 2023

infinisil mentioned this pull request Sep 25, 2023

[WIP] File set combinators #222981

Closed

7 tasks

ofborg bot added 10.rebuild-darwin: 0 10.rebuild-linux: 1-10 labels Sep 26, 2023

infinisil mentioned this pull request Sep 26, 2023

lib.fileset: Representation for empty file sets without a base path #257351

Merged

2 tasks

infinisil force-pushed the fileset.intersect branch 2 times, most recently from cc723f2 to 98346b0 Compare October 3, 2023 22:27

infinisil changed the title ~~lib.fileset,intersect, lib.fileset.intersects: init~~ lib.fileset,intersection: init Oct 3, 2023

infinisil changed the title ~~lib.fileset,intersection: init~~ lib.fileset.intersection: init Oct 3, 2023

infinisil commented Oct 3, 2023

View reviewed changes

infinisil force-pushed the fileset.intersect branch 3 times, most recently from 10a4700 to 81bb565 Compare October 4, 2023 19:35

infinisil marked this pull request as ready for review October 4, 2023 19:36

infinisil requested review from roberth and fricklerhandwerk October 4, 2023 19:36

lib.fileset: Refactor for performance and future re-use

2541635

infinisil force-pushed the fileset.intersect branch from 81bb565 to 86b6d46 Compare October 4, 2023 21:22

fricklerhandwerk approved these changes Oct 5, 2023

View reviewed changes

delroth added the 12.approvals: 1 label Oct 8, 2023

roberth reviewed Oct 11, 2023

View reviewed changes

lib/fileset/README.md Outdated Show resolved Hide resolved

roberth reviewed Oct 11, 2023

View reviewed changes

roberth approved these changes Oct 11, 2023

View reviewed changes

delroth added 12.approvals: 2 and removed 12.approvals: 1 labels Oct 11, 2023

infinisil and others added 2 commits October 11, 2023 16:17

lib.fileset.intersection: init

4ecf025

lib.fileset: Minor contributor doc adjustments

389be8d

Co-authored-by: Robert Hensing <robert@roberthensing.nl>

infinisil force-pushed the fileset.intersect branch from 86b6d46 to 389be8d Compare October 11, 2023 14:18

delroth removed the 12.approvals: 2 label Oct 11, 2023

infinisil merged commit 006b28f into NixOS:master Oct 11, 2023
19 checks passed

infinisil deleted the fileset.intersect branch October 11, 2023 15:33

infinisil mentioned this pull request Nov 9, 2023

File set library tracking issue and feature requests #266356

Open

infinisil mentioned this pull request Nov 14, 2023

fileset.fileFilter: Restrict second argument to paths #267384

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`lib.fileset.intersection`: init #257356

`lib.fileset.intersection`: init #257356

infinisil commented Sep 26, 2023 •

edited

infinisil Oct 3, 2023

fricklerhandwerk Oct 4, 2023

roberth Oct 11, 2023

infinisil commented Oct 4, 2023

infinisil commented Oct 10, 2023

roberth Oct 11, 2023

infinisil Oct 11, 2023

roberth left a comment

roberth Oct 11, 2023

roberth Oct 11, 2023

roberth Oct 11, 2023

infinisil Oct 11, 2023

lib.fileset.intersection: init #257356

lib.fileset.intersection: init #257356

Conversation

infinisil commented Sep 26, 2023 • edited

Description of changes

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

infinisil commented Oct 4, 2023

infinisil commented Oct 10, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roberth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

`lib.fileset.intersection`: init #257356

`lib.fileset.intersection`: init #257356

infinisil commented Sep 26, 2023 •

edited