Use new build meta-data #636
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gctucker
force-pushed
the
bmeta-rework
branch
9 times, most recently
from
1d074e9
to
961e70f
Compare
mgalka
suggested changes
Apr 16, 2021
gctucker
force-pushed
the
bmeta-rework
branch
3 times, most recently
from
6fccc28
to
dbd23d1
Compare
Add optional arguments to manually provide the Git commit, describe and describe "verbose" to RevisionData.run(). This is primarily useful when building a kernel source tree that is not a Git repository, such as a source tarball. Update command line arguments in "kci_build init_bmeta" accordingly. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Pass the YAML configs data to MakeStep._run_step() as they may need it when running steps. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Expand the fragment names to their full path in MakeConfig when they are listed in the YAML configuration. This allows things like defconfig+kselftest to be used from a user point of view, as it was already the case before the refactoring with steps. Then the expanded defconfig will have defconfig+kernel/configs/kselftest.config with the actual path to the fragment file. Both versions (full and expanded) are stored in bmeta.json. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add the kernel publish_path to the meta-data in MakeConfig. This will be needed when pushing build artifacts to the storage server. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add kernelci.storage.discover_files() to find files before sending them to the storage server with upload_files(). Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add kernelci.build.MetaStep as a Step class solely for the purpose of using the meta-data from the previously saved JSON files. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Update the "kci_build push_kernel" implementation to use the new kernelci.build.Step class and associated meta-data. Also use the code in kernelci.storage to send the artifacts to drop the dependenty from kernelci.build. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Remove the hardcoded '/test' path in the database API URLs as kci_data is intended to be used to send any kind of data, not just test results. Update kci_data and db-configs.yaml accordingly. Also update docstrings and copyright notices. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Rather than providing the path where to send the data, make it part of
the data itself and leave it up to each database implementation to
deal with it accordingly. Test results now need to wrap all the data
with "test": {}. Update kernelci_backend accordingly.
Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Rather than raw JSON, accept a Python dictionary with the Database.submit() method. This drops the dependency on JSON from the abstract class, so the calling code can construct the data payload arbitrarily. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add the Database.submit_build() method as a special entry point to pass a kernelci.build.MetaStep object and submit the kernel build meta-data to the database. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add a Database.submit_test() method as an alternative entry point to the generic .submit() method. Similarly to .submit_build(), the .submit_test() method is to send one set of test results using the same schema as previously used with .submit() when it was hard-coded for test results only. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
To avoid ambiguity, rename the "describe_v" revison field to "describe_verbose". This probably should eventually be dropped to just have "describe", when reviewing the schema. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Rework how the artifacts.json file is being handled and the schema it uses so that it can be actually consumed into the database: * create one entry for each step at the top-level * add artifacts types: file, directory, tarball * add a "path" attribute to each entry so they can be found, relative to the _install_ directory * add an optional "key" attribute to be able to find artifacts * add an optional "contents" attribute as a list of artifacts that can be found in directories and tarballs * update all the parts of the code that add new artifacts accordingly Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add "steps" and "artifacts" to the bmeta data structure in MetaStep so it can all be accessed within a single namespace. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Rather than creating a new JSON data structure for the HTTP POST request when publishing a build, and then having the backend rely on the bmeta.json file previously updated to find extra pieces of data, send the entire meta-data structure. This depends on the backend code to be updated accordingly, to validate and parse this data. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Print any HTTP errors when sending requests to the backend. This is particularly useful when the backend is doing the JSON schema validation and sending errors back to the client. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Update the job-build.jinja2 Kubernetes template with the new kci_build commands to build kernels in separate steps. Generate a local config file to simplify command line arguments and keep the API token secret when sending data to the backend. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
When listing all the combinations to build, use the short fragment name rather than the expanded one. The expanded one will be determined later when doing the actual kernel build. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Fix a mistake in the call to add the build step meta-data which resulted in the wrong status to be saved. Fixes: fa346b3 ("kernelci.build: use Step.name in meta-data") Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add properties and class methods to define the default build output and install paths. Use them in the Step() class constructor. These can be used by client code to determine where the files will be created by default. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Drop the MetaStep class and create the Metadata class instead to hold all the meta-data implementation independently from the Step implementation. Use this class inside Step to handle meta-data. Client code that needs to read the meta-data can just create a Metadata class instead of MetaStep which was accidentally also a build step. Update all the build steps to use the new Metadata class. The build meta-data is now accessible via the 'bmeta' key, similarly to 'steps' and 'artifact'. The artifacts are now directly handled exactly like stored in artifacts.json and an internal representation using a dictionary is now kept in self._artifacts_map to simplify the logic when adding artifacts. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Fix the message about where the build meta-data is being initialised by using the output path actually defined in the build step rather than the command line argument which may be None. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Use the new Step.get_install_path() method to get the kernel build installation path reliably following the default values. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Use the new Metadata.get() method rather than get_value() when getting the whole build meta-data to submit to the database for a kernel build. Update the docstrings in kernelci.data.Database accordingly to the new kernelci.build.Metadata class. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Add the Args.build_output command line argument to differentiate the test output directory from the kernel build output one. This is primarily useful when reading meta-data from the kernel build output and writing job definitions in a separate output directory. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Use the new kenelci.build.Metadata class instead of MetaStep to get the build meta-data. Use the new kernelci.build.Step.get_install_path method to get the kernel installation path using the conventional default values. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Align the signatures for the Step.run() method in all the derived classes using an "opts" argument as a dictionary with arbitrary options for each step. This makes the implementation easier to follow with a cleaner design. Update kci_build accordingly. Signed-off-by: Guillaume Tucker <guillaume.tucker@collabora.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Full integration of the new build meta-data with k8s builds and generated LAVA jobs.
Should be merged at the same time as the following PRs: