Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sourcepolicy #3332

Merged
merged 15 commits into from
Dec 13, 2022
Merged

Sourcepolicy #3332

merged 15 commits into from
Dec 13, 2022

Conversation

cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Nov 28, 2022 •
edited

Add source policy in LLB as described in #2943 (comment)

Replaces #2943
Closes #2943

@cpuguy83 cpuguy83 force-pushed the sourcepolicy branch 3 times, most recently from e8f8a03 to 4e308ac Compare November 28, 2022 23:20
tonistiigi
tonistiigi reviewed Nov 29, 2022
View reviewed changes
docs/build-repro.md Outdated Show resolved Hide resolved
@tonistiigi tonistiigi added this to the v0.11.0 milestone Nov 29, 2022
@cpuguy83 cpuguy83 force-pushed the sourcepolicy branch 6 times, most recently from 1ad024c to 43e8ffb Compare November 29, 2022 22:32
@cpuguy83 cpuguy83 marked this pull request as ready for review November 29, 2022 22:49
@cpuguy83
Copy link
Member Author

Marking this ready, looks like it's gonna be all 💚

jedevc
jedevc reviewed Nov 30, 2022
View reviewed changes
sourcepolicy/matcher.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
sourcepolicy/engine_test.go Outdated Show resolved Hide resolved
AkihiroSuda
AkihiroSuda reviewed Nov 30, 2022
View reviewed changes
solver/llbsolver/solver.go Outdated
@@ -125,7 +129,7 @@ func (s *Solver) Bridge(b solver.Builder) frontend.FrontendLLBBridge {
return s.bridge(b)
}

func (s *Solver) Solve(ctx context.Context, id string, sessionID string, req frontend.SolveRequest, exp ExporterRequest, ent []entitlements.Entitlement, post []Processor) (_ *client.SolveResponse, err error) {
func (s *Solver) Solve(ctx context.Context, id string, sessionID string, req frontend.SolveRequest, exp ExporterRequest, ent []entitlements.Entitlement, post []Processor, srcPol *sourcepolicypb.Policy) (_ *client.SolveResponse, err error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SolverResponse should contain a (boolean?) value that indicates whether the policy was consumed or not

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consumed in what way? Policy applied to sources?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a boolean here is not very useful except for just 1-off testing cases.
Probably it would be best to look at the build metadata to see what sources were actually used... maybe we could add something to trace policy usage in that metadata, but this will require a bit more effort.

AkihiroSuda
AkihiroSuda reviewed Nov 30, 2022
View reviewed changes
docs/build-repro.md Outdated Show resolved Hide resolved
@cpuguy83 cpuguy83 force-pushed the sourcepolicy branch 3 times, most recently from f491e5f to 9502353 Compare December 2, 2022 19:04
@cpuguy83
Copy link
Member Author

cpuguy83 commented Dec 2, 2022

This is updated with suggestions.

@cpuguy83 cpuguy83 force-pushed the sourcepolicy branch 2 times, most recently from 32a6668 to 2bce7d0 Compare December 2, 2022 20:02
jedevc
jedevc reviewed Dec 5, 2022
View reviewed changes
sourcepolicy/engine.go Outdated Show resolved Hide resolved
tonistiigi
tonistiigi reviewed Dec 8, 2022
View reviewed changes
sourcepolicy/pb/policy.proto Outdated Show resolved Hide resolved
sourcepolicy/pb/policy.proto Outdated Show resolved Hide resolved
sourcepolicy/matcher.go Outdated Show resolved Hide resolved
sourcepolicy/matcher.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
docs/build-repro.md Outdated Show resolved Hide resolved
docs/build-repro.md Outdated Show resolved Hide resolved
hack/dockerfiles/generated-files.Dockerfile Outdated Show resolved Hide resolved
solver/llbsolver/vertex.go Outdated Show resolved Hide resolved
sourcepolicy/engine.go Outdated Show resolved Hide resolved
tonistiigi
tonistiigi reviewed Dec 13, 2022
View reviewed changes
sourcepolicy/engine.go Outdated Show resolved Hide resolved
tonistiigi
tonistiigi reviewed Dec 13, 2022
View reviewed changes
sourcepolicy/engine.go Outdated Show resolved Hide resolved
tonistiigi
tonistiigi reviewed Dec 13, 2022
View reviewed changes
sourcepolicy/engine.go Outdated Show resolved Hide resolved
@cpuguy83 cpuguy83 force-pushed the sourcepolicy branch 2 times, most recently from 87725f9 to 385f4f0 Compare December 13, 2022 20:14
tonistiigi
tonistiigi reviewed Dec 13, 2022
View reviewed changes
sourcepolicy/engine.go Outdated Show resolved Hide resolved
AkihiroSuda and others added 15 commits December 13, 2022 13:01
@AkihiroSuda @tonistiigi
llbsolver: support pinning sources
0d3f71e 
Alternative to PR 2816 ("dockerfile: support Dockerfile.pin for pinning sources")

This version is implemented on the llbsolver side and agnostic to the LLB frontends.
See `solver/llbsolver/vertex.go:loadLLB()`.

See `docs/build-repro.md` for the usage.

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
@cpuguy83 @tonistiigi
Update source pinning to use proposed policy
7526659 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @AkihiroSuda @tonistiigi
sourcepolicy: add wildcard match type
51ad0c7 
This defines a better type for wildcard matching as well as a more
appropritate package (contributed by Akihiro) for doing wildcard
matching.

Also some improvements to caching state such as compiled regexes.

Co-authored-by: Akihiro Suda <suda.kyoto@gmail.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: split dest type from identifier
6e89b21 
This makes destination more symetrical with sources.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: Add support for frontend policies
7a08c9d 
Allows frontends to pass in a list of policies in the solve request.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: cleanup uneccessary interfaces
c7a54c7 
These were just there while figuring out how things would work and are
not needed now.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: Do not pass state between policies
3e93b54 
Per our discussion on github, each policy should be evaluated on it's
own.
ie. an "allow" in one policy should be able to change to a "deny" in
another policy.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
Fix some typos
1893912 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
Recompute all digests recursively when mutated
4bc6694 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: extra test
e4ff85a 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: remove evalState
b6db53a 
This makes it so a deny is always a deny regardless of if it was allowed
in a prior rule.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
Simplify Op digest recompute
bf809b7 
This turns the closure into a proper recursive funciton.
It also ensures there isn't a chance of infinite recursion due to
changes to op inputs.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>

fix
@cpuguy83 @tonistiigi
sourcepolicy: Change field names and collapse type
6d061b9 
Updates the field names for source policies to be a little less
confusing: `Source` -> `Selector`, `Destination` -> `Updates`.

Also collapse `Type` into the `Identifier` which matches how the rest of
buildkit works and greatly simplifies matching.. and generally makes it
so the policy engine doesn't need to care about schemes.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
doc: build-repro: Any source can be made reproducible
adea592 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@cpuguy83 @tonistiigi
sourcepolicy: fix policy so last rule wins
c40f30e 
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
@tonistiigi tonistiigi merged commit a489366 into moby:master Dec 13, 2022
@cpuguy83 cpuguy83 deleted the sourcepolicy branch December 14, 2022 00:00
@jlongtine jlongtine mentioned this pull request Dec 16, 2022
Open
@mircubed mircubed mentioned this pull request Mar 1, 2023
Open
@tianon tianon mentioned this pull request May 26, 2023
Closed
@tonistiigi tonistiigi mentioned this pull request May 6, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@crazy-max crazy-max crazy-max left review comments

@jedevc jedevc jedevc left review comments

@AkihiroSuda AkihiroSuda AkihiroSuda left review comments

@tonistiigi tonistiigi tonistiigi approved these changes

Assignees
No one assigned
Labels
kind/enhancement
Projects
None yet
Milestone
v0.11.0
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@cpuguy83 @tonistiigi @crazy-max @jedevc @AkihiroSuda