-
Notifications
You must be signed in to change notification settings - Fork 96
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
unawareness runtime prefetch #572
base: main
Are you sure you want to change the base?
Conversation
billie60
commented
Jan 5, 2024
- add prefetchlist store prefetchlist storage service.
- Modify the optimizer to publish the access file list as a prefetchlist to the storage service when obtaining it.
059d595
to
2994ad5
Compare
@@ -0,0 +1,96 @@ | |||
/* |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about naming it tools/prefetch-distribution
?
func main() { | ||
serverCache = Cache{Items: make(map[string]*CacheItem)} | ||
|
||
http.HandleFunc("/api/v1/prefetch/upload", uploadHandler) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should only accept POST for upload
and GET for download
, we'd better use the echo
web framework.
@@ -104,6 +109,16 @@ func buildFlags(args *PluginArgs) []cli.Flag { | |||
Usage: "whether to overwrite the existed persistent files", | |||
Destination: &args.Config.Overwrite, | |||
}, | |||
&cli.StringFlag{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
&cli.StringFlag{
Name: "prefetch-distribution-endpoint",
Usage: "The service endpoint of prefetch distribution, for example: http://localhost:1323/api/v1/prefetch/upload",
Destination: &args.Config.PrefetchDistributionEndpoint,
},
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any updates?
cmd/optimizer-nri-plugin/main.go
Outdated
func (p *plugin) StartContainer(_ *api.PodSandbox, container *api.Container) error { | ||
dir, imageName, err := GetImageName(container.Annotations) | ||
if err != nil { | ||
return err | ||
} | ||
containerName := container.Annotations[containerNameLabel] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Does the annotation exist?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, even without adding annotations in container.yaml, it can still use container.Annotations [containerNameLabel]
to get container name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you only need container name
, container.Name
is enough.
log.WithError(err).Error("failed to send prefetch to http server") | ||
} | ||
}() | ||
|
||
globalFanotifyServer[imageName] = fanotifyServer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should protect the map using mutex.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any updates?
cmd/optimizer-nri-plugin/main.go
Outdated
|
||
err = postRequest(item, url) | ||
if err != nil { | ||
return fmt.Errorf("error uploading to server: %w", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use errors.Wrap
.
cmd/optimizer-nri-plugin/main.go
Outdated
func sendToServer(imageName, prefetchlistPath, containerName, serverURL string) error { | ||
data, err := os.ReadFile(prefetchlistPath) | ||
if err != nil { | ||
return fmt.Errorf("error reading file: %w", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use errors.Wrap
.
cmd/optimizer-nri-plugin/main.go
Outdated
return fmt.Errorf("failed to read response body: %w", err) | ||
} | ||
|
||
fmt.Println("Server Response:", string(body)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should check the HTTP status code and return an error if the status code is invalid.
bed33a8
to
01d5e46
Compare
cmd/optimizer-nri-plugin/main.go
Outdated
func (p *plugin) StartContainer(_ *api.PodSandbox, container *api.Container) error { | ||
dir, imageName, err := GetImageName(container.Annotations) | ||
if err != nil { | ||
return err | ||
} | ||
containerName := container.Annotations[containerNameLabel] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you only need container name
, container.Name
is enough.
} | ||
} | ||
|
||
item := CacheItem{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we define the generic API for different storage service? Sometimes, we may need to send more information to the server.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also, does the API provide a query endpoint?
return c.String(http.StatusBadRequest, "Invalid request payload") | ||
} | ||
|
||
serverCache.Set(item) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we define an interface for difference storage backend?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess can we move the fanotify
server to an separate component. Because for containerd 1.6.x
, the NRI runs in a different construct. If we do this, the NRI for both containerd versions can be simplified to send a request to server
.
d0582fa
to
b96cf48
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #572 +/- ##
==========================================
+ Coverage 34.64% 34.66% +0.01%
==========================================
Files 65 65
Lines 6552 6552
==========================================
+ Hits 2270 2271 +1
+ Misses 3967 3966 -1
Partials 315 315
|
4111e29
to
dcd0f2f
Compare
bf97c9a
to
1fd5320
Compare
546b4d2
to
c2fa245
Compare
.github/workflows/optimizer.yml
Outdated
sudo systemctl restart nydus-snapshotter | ||
fi | ||
sleep 5 | ||
NYDUS_VER=v$(curl --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}' -s "https://api.github.com/repos/dragonflyoss/nydus/releases/latest" | jq -r .tag_name | sed 's/^v//') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The --header 'authorization: Bearer ${{ secrets.GITHUB_TOKEN }}'
can be removed.
.github/workflows/optimizer.yml
Outdated
echo "Error: --prefetch-files not found in nydus-snapshotter.log" | ||
exit 1 | ||
fi | ||
# output=$(sudo cat /etc/nydus/logs/nydus-snapshotter.log | grep "nydusd command") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be removed?
@@ -104,6 +109,16 @@ func buildFlags(args *PluginArgs) []cli.Flag { | |||
Usage: "whether to overwrite the existed persistent files", | |||
Destination: &args.Config.Overwrite, | |||
}, | |||
&cli.StringFlag{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any updates?
log.WithError(err).Error("failed to send prefetch to http server") | ||
} | ||
}() | ||
|
||
globalFanotifyServer[imageName] = fanotifyServer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Any updates?
router *mux.Router | ||
} | ||
|
||
const endpointImageName string = "/api/v1/imagename" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
/api/v1/prefetch?reference=$image
} | ||
defer resp.Body.Close() | ||
|
||
responseBody, err := io.ReadAll(resp.Body) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should check if resp.StatusCode != http.StatusOK {
before this.
go.mod
Outdated
@@ -55,6 +56,12 @@ require ( | |||
k8s.io/utils v0.0.0-20230726121419-3b25d923346b | |||
) | |||
|
|||
require ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merge these into below.
@@ -0,0 +1,135 @@ | |||
version = 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we reuse misc/snapshotter/config.toml
instead of creating a new one?
dde1a0c
to
ce605e9
Compare
1. add prefetchlist store prefetchlist storage service. 2. Modify the optimizer to publish the access file list as a prefetchlist to the storage service when obtaining it. 3. modify http server add lru algo 4. use echo web framework 5. modify based on comments 6. get and post prefetchlist in optimizer
1. send post request to optimizer 2. store prefetchlist 3. add prefetchlist in nydusd
ce605e9
to
394a502
Compare