-
Notifications
You must be signed in to change notification settings - Fork 0
Github mirror of "operations/software/elasticsearch/plugins" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing
wikimedia/operations-software-elasticsearch-plugins
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Debian package to bundle elasticsearch plugins needed for search. I. PREPARE A NEW VERSION ------------------------ When adding a new plugin or upgrading to a new elastic version the developer needs to: - update debian/rules to set the new elastic/build version - update debian/plugin_urls.lst to add new plugins or change custom versions make sure the GPG key ID (separated by comma at the end of a line) matches the one used for signing the release, and that this key has been published - update debian/changelog to indicate what's new (`dch -i`). Then they need to generate the debian/sha256sums file: $ ./debian/rules prepare_commit This command will execute the bash script debian/checkout.bash. This script will: - download all plugins listed in debian/plugin_urls.lst - verify gpg signature if one is provided - check for elasticsearch version - assemble the plugin files to debian/blobs - generate a new debian/sha256sums If debian/blobs looks good the developer can git commit -a && git review (NOTE that debian/blobs is never uploaded to gerrit) II. REVIEW A PATCH ----------------- The process relies on the fact that at least two engineers will fetch the plugins from their source. The debian/sha256sums file will serve as a verification to make sure that the build is repeatable and that no binary blob was uploaded maliciously. To review they must simply run: $ git review -d gerrit_id $ ./debian/rules verify_commit It will perform exactly the same operations as prepare_commit except that the debian/sha256sums file is not generated. The sums are validated. They can then verify that debian/blobs looks good. If everything is OK the reviewer can +2 the patch. III. BUILD THE DEBIAN PACKAGE --------------------------- An ops engineer can then build the package. First they need to fetch the blobs on their local machine prior to uploading to the active package build server: $ ./debian/rules prepare_build It'll download all the files and make all the verifications again. If everything looks good they can scp the whole dir to the active package build server and then build the package with $ DIST=bullseye-wikimedia pdebuild NOTE: the sha256sums are verified again. The package can then be uploaded to the apt repository. IV. BLOBS VERIFICATION ---------------------- In the end the blobs will be download 3 times (at least 2 if ops == reviewer): 1. The gerrit uploader 2. The reviewer 3. The ops GPG signatures will be checked 3 times (at least 2 if ops == reviewer): 1. The gerrit uploader 2. The reviewer 3. The ops The sha256sums file is generated only one time per patch by the gerrit uploader. It then verified 3 times: 1. The reviewer 2. The ops on the prepare_build command 3. On the active package build server by dpkg-buildpackage V. FILES --------- debian/rules Makefile for dh and custom build steps debian/sha256sums List of all sha256 sums that were generated by the last prepare_commit debian/plugin_urls.lst List of plugins to install, the format is as follow: debian/changelog Debian changelog file debian/checkout.bash Bash script to fetch&verify plugins from their source VI. FORMATS ----------- plugin_urls.lst is a flat file with one line per plugin formatted as follow: URL,GPG_KEY e.g. https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-icu/analysis-icu-$ELASTICSEARCH_VERSION.zip,D27D666CD88E42B4 Use none when the zip is not signed. $ELASTICSEARCH_VERSION will resolve to the version defined in debian/rules, it's useful for core plugins. VII. DEVELOPER IMAGES --------------------- When shipping a new version of the plugins you might want to also update the images used by developers that work with CirrusSearch and Elasticsearch. The project is named cirrussearch-elasticsearch-image (https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-elasticsearch-image).
About
Github mirror of "operations/software/elasticsearch/plugins" - our actual code is hosted with Gerrit (please see https://www.mediawiki.org/wiki/Developer_access for contributing
Resources
Code of conduct
Stars
Watchers
Forks
Releases
No releases published
Packages 0
No packages published