Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a way to ignore portions of the signature for cache key calcu… #4324

Closed
wants to merge 4 commits into from

Conversation

troychiu
Copy link
Member

Tracking issue

Closes #3058

Describe your changes

  1. add cache_ignore_input_vars as a field of TaskMetadata
  2. change the signature of GenerateArtifactTagName to GenerateArtifactTagName(ctx context.Context, inputs *core.LiteralMap, cacheIgnoreInputVars *[]string)
  3. when cacheIgnoreInputVars provided, skip all input variables in it for hash calculation

Check all the applicable boxes

  • I updated the documentation accordingly.
  • All new and existing tests passed.
  • All commits are signed-off.

Screenshots

Note to reviewers

The local version is here: flyteorg/flytekit#1914

…lation purposes

Signed-off-by: troychiu <y.troychiu@gmail.com>
@codecov
Copy link

codecov bot commented Oct 30, 2023

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (956b24e) 59.81% compared to head (30d534c) 59.83%.

Files Patch % Lines
flytepropeller/pkg/controller/nodes/task/cache.go 0.00% 5 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4324      +/-   ##
==========================================
+ Coverage   59.81%   59.83%   +0.01%     
==========================================
  Files         632      632              
  Lines       53644    53655      +11     
==========================================
+ Hits        32089    32102      +13     
+ Misses      19029    19028       -1     
+ Partials     2526     2525       -1     
Flag Coverage Δ
unittests 59.83% <77.27%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: troychiu <y.troychiu@gmail.com>
Comment on lines +120 to +131
var inputsAfterIgnore *core.LiteralMap
if cacheIgnoreInputVars != nil {
inputsAfterIgnore = &core.LiteralMap{Literals: make(map[string]*core.Literal)}
for name, literal := range inputs.Literals {
if slices.Contains(*cacheIgnoreInputVars, name) {
continue
}
inputsAfterIgnore.Literals[name] = literal
}
} else {
inputsAfterIgnore = inputs
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than creating a new LiteralMap here would it make more sense to pass the cacheIgnoreInputVars to the HashLiteralMap function. Then in this code here we just need to update to:

	for name, literal := range literalMap.Literals {
	        if !slices.Contains(cacheIgnoreInputVars, name) {
		    hashifiedLiteralMap[name] = hashify(literal)
		}
	}

This achieves the same outcome in 2 LoC rather than 12. I think it makes more sense to only create a new LiteralMap in one place as well.

@hamersaw hamersaw mentioned this pull request Dec 18, 2023
3 tasks
@hamersaw
Copy link
Contributor

Closing as satisfied by #4618

@hamersaw hamersaw closed this Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core feature] Provide a way to ignore portions of the signature for cache key calculation purposes
4 participants