Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/build-extensions-container: use cosa meta to update meta.json #3152

Merged
merged 1 commit into from
Nov 2, 2022

Conversation

jmarrero
Copy link
Member

closes #3149

@dustymabe
Copy link
Member

dustymabe commented Oct 31, 2022

For some reason I was thinking there would be a way for us to pass in just the new json (for the extensions container) and have that all get merged properly. I think what we have here has the same problems as before (i.e. we read the meta.json and later call a program that updates it). There is time between the read and cosa meta call that other things could have happened.

Maybe the cosa meta code that merges things underneath has logic that handles this properly.

@jlebon
Copy link
Member

jlebon commented Oct 31, 2022

Maybe the cosa meta code that merges things underneath has logic that handles this properly.

Yup, that's correct. See the write_json() function in cmdlib.py which reads the meta.json, merges in the modifications, and writes it back out, all under lock. One thing to note though is that the coreos-assembler.meta-stamp key needs to be bumped so that the merge function knows something actually changed.

@@ -68,7 +68,7 @@ func buildExtensionContainer() error {
metapath := filepath.Join(buildPath, "meta.json")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of reading in the meta.json here and defining a new var cosaBuild cosa.Build below, could we just use the lastBuild that we already have from the cosa.ReadBuild() call at the beginning of this function?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we do do that I wouldn't mind renaming lastBuild to cosaBuild above and using that variable name.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for implementing this. I do realize now though that there is a lot more time that passes from the time we read the build (and current meta.json) and we write it out. I think that's OK, just wanted to call it out in case this raises any alarm bells with @jlebon.

@dustymabe
Copy link
Member

Maybe the cosa meta code that merges things underneath has logic that handles this properly.

Yup, that's correct. See the write_json() function in cmdlib.py which reads the meta.json, merges in the modifications, and writes it back out, all under lock. One thing to note though is that the coreos-assembler.meta-stamp key needs to be bumped so that the merge function knows something actually changed.

👍 - so if we update cosaBuild.MetaStamp then we should be good?

@jmarrero jmarrero force-pushed the lock-meta branch 2 times, most recently from 31cd31a to e69965c Compare November 1, 2022 14:06
cosaBuild.BuildArtifacts.ExtensionsContainer = &cosa.Artifact{
Path: targetname,
Sha256: sha256sum,
SizeInBytes: float64(stat.Size()),
SkipCompression: true,
}
cosaBuild.MetaStamp = cosaBuild.MetaStamp + 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jlebon is this safe or should we go with the current time?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd still work for our purposes, but yeah ideally it should be the current time. This should work:

Suggested change
cosaBuild.MetaStamp = cosaBuild.MetaStamp + 1
cosaBuild.MetaStamp = time.Now().UnixNano()

cosaBuild.BuildArtifacts.ExtensionsContainer = &cosa.Artifact{
Path: targetname,
Sha256: sha256sum,
SizeInBytes: float64(stat.Size()),
SkipCompression: true,
}
cosaBuild.MetaStamp = cosaBuild.MetaStamp + 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd still work for our purposes, but yeah ideally it should be the current time. This should work:

Suggested change
cosaBuild.MetaStamp = cosaBuild.MetaStamp + 1
cosaBuild.MetaStamp = time.Now().UnixNano()

cmd/build-extensions-container.go Outdated Show resolved Hide resolved
@cgwalters
Copy link
Member

I guess we've been taking a hit here in blazing a trail in using Go for cosa build stuff. Hopefully it will pay itself back later by writing less shell script (and involving less dynamic python tracebacks)...

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good - one more round and I think we can merge!

@jmarrero
Copy link
Member Author

jmarrero commented Nov 2, 2022

/test rhcos


err = ioutil.WriteFile(metapath, newBytes, 0644)
extensions_container_meta_path := filepath.Join(buildPath, "meta.extensions-container.json")
err = ioutil.WriteFile(extensions_container_meta_path, newBytes, 0644)
if err != nil {
return err
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're missing a defer here to delete that file, no?

@cgwalters
Copy link
Member

Related to sharing locking; I think it would make sense to do the lowest common denominator of forking off /usr/bin/flock as a separate subprocess. That would help ensure we have a consistent flow across shell script, Python and Go.

Copy link
Member

@dustymabe dustymabe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dustymabe dustymabe enabled auto-merge (rebase) November 2, 2022 16:12
@dustymabe dustymabe merged commit 648414c into coreos:main Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add locking to golang build-extensions-container
4 participants