Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added checksum=None
to the test_dbg
artifact test to make it pass. Overall, looks great. Absolutely love the parallelization!
rhcephcompose/compose.py
Outdated
@@ -48,6 +48,8 @@ def __init__(self, conf): | |||
self.extra_files = conf['extra_files'] | |||
# Whether sources composition should be skipped | |||
self.include_sources = conf.get('include_sources', True) | |||
# Whether -dbg composition should be skipped |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just noticed this comment. It describes the opposite of the behavior; it should probably be:
Whether -dbg composition should happen
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're right, will rebase this branch and fix
5f4332e
to
2c245c5
Compare
Add a new property to BinaryArtifact objects, "dbg_parent". This will let us identify binaries as "-dbg" or not, and provide us with the binary's parent name. The purpose of this change is to expose this data to callers (Compose class). Eventually we can check this parent name against our comps manifest. This will allow us to add each package's -dbg bits into a compose without having to list each "foo-dbg" and "foo-dbg" package pair explicitly in the comps .xml manifest.
If a BuildArtifact is a -dbg package, and its parent package is listed in a comps manifest, add that -dbg package to the compose. The purpose of this change is to ship -dbg packages in the product deliverable for customers. Use a separate -dbg directory so we don't duplicate so much data. RHCS 2 is about 8GB when we copy the dbg packages into each variant, and only 2.3GB when we route everything to this single -dbg directory. Add a "include_dbg" configuration option to control this -dbg dir (useful for speeding up test composes prior to GA). A future change could turn this -dbg directory into a proper repository.
I've dropped the commit that runs each variant in its own PID, because they were racing while downloading/using files from the .cache location (sha512 checksum failures). We'll have to optimize the performance with more testing in a future PR. |
This change needs further work and testing before we merge this.
With this change, rhcephcompose will place all of the debug packages into a separate "-debug" directory.
A new setting
include_dbg
toggles this behavior. (We'll want to disableinclude_dbg
in Jenkins CI runs in order to save time and disk space).TODO:
generate a deb repository with reprepro in the "-debug" directory, rather than simply dumping all the files into that location.
Fix tests so they pass
More testing to ensure this does not cause regressions
https://bugzilla.redhat.com/1375078