Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include migration builds in image builds #716

Merged
merged 2 commits into from
Feb 14, 2020
Merged

Include migration builds in image builds #716

merged 2 commits into from
Feb 14, 2020

Conversation

zmrow
Copy link
Contributor

@zmrow zmrow commented Feb 7, 2020

Issue #, if available:
Related to #606

Description of changes:
This change includes migration builds with image builds. It
will build all of the migrations, strip them, lz4 compress them, and
tar them up as an output of the build.

Testing done:

  • Built a working AMI
  • cargo make outputs expected artifacts and migrations
[mrowicki@ip-10-0-60-75 PRIVATE-thar]$ ls build/*migrations*
build/thar-x86_64-aws-k8s-0.2.2-7f117d5a-dirty-migrations.tar  build/thar-x86_64-migrations-debuginfo-0.0-0.x86_64.rpm
build/thar-x86_64-migrations-0.0-0.x86_64.rpm

[mrowicki@ip-10-0-60-75 PRIVATE-thar]$ tar tf build/thar-x86_64-aws-k8s-0.2.2-7f117d5a-dirty-migrations.tar
migrate_v0.1_borkseed.lz4
migrate_v0.1_host-containers-version.lz4
migrate_v0.2_containerd-config-path.lz4
migrate_v0.2_remove-region.lz4

[mrowicki@ip-10-0-60-75 PRIVATE-thar]$ cd build/
[mrowicki@ip-10-0-60-75 build]$ tar xf thar-x86_64-aws-k8s-0.2.2-7f117d5a-dirty-migrations.tar
[mrowicki@ip-10-0-60-75 build]$ ls migrate*
migrate_v0.1_borkseed.lz4  migrate_v0.1_host-containers-version.lz4  migrate_v0.2_containerd-config-path.lz4  migrate_v0.2_remove-region.lz4

[mrowicki@ip-10-0-60-75 build]$ file migrate_v0.1_borkseed.lz4 
migrate_v0.1_borkseed.lz4: LZ4 compressed data (v1.4+)

[mrowicki@ip-10-0-60-75 build]$ lz4 -d migrate_v0.1_borkseed.lz4
Decoding file migrate_v0.1_borkseed 
migrate_v0.1_borksee : decoded 558776 bytes   
                                 
[mrowicki@ip-10-0-60-75 build]$ ls migrate*
migrate_v0.1_borkseed  migrate_v0.1_borkseed.lz4  migrate_v0.1_host-containers-version.lz4  migrate_v0.2_containerd-config-path.lz4  migrate_v0.2_remove-region.lz4

[mrowicki@ip-10-0-60-75 build]$ file migrate_v0.1_borkseed
migrate_v0.1_borkseed: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=596b4f3b56938d6d86104ba228ce83266b806fa6, stripped

[mrowicki@ip-10-0-60-75 build]$ ls -lah migrate*
-r-xr-xr-x. 1 mrowicki mrowicki 546K Feb 13 17:05 migrate_v0.1_borkseed
-r-xr-xr-x. 1 mrowicki mrowicki 353K Feb 13 17:05 migrate_v0.1_borkseed.lz4
-r-xr-xr-x. 1 mrowicki mrowicki 333K Feb 13 17:05 migrate_v0.1_host-containers-version.lz4
-r-xr-xr-x. 1 mrowicki mrowicki 332K Feb 13 17:05 migrate_v0.2_containerd-config-path.lz4
-r-xr-xr-x. 1 mrowicki mrowicki 345K Feb 13 17:05 migrate_v0.2_remove-region.lz4

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

macros/aarch64 Outdated Show resolved Hide resolved
packages/migrations/migrations.spec Outdated Show resolved Hide resolved
packages/migrations/migrations.spec Outdated Show resolved Hide resolved
packages/migrations/migrations.spec Outdated Show resolved Hide resolved
Makefile.toml Outdated Show resolved Hide resolved
@zmrow
Copy link
Contributor Author

zmrow commented Feb 7, 2020

Removed Ben's commit

@zmrow zmrow force-pushed the make-migrations branch 2 times, most recently from fddc726 to 5aa667e Compare February 7, 2020 17:52
@zmrow
Copy link
Contributor Author

zmrow commented Feb 7, 2020

Updated this to remove the migrations package. The way I had it set up it was building many expensive things twice, for no real benefit.

This change rolls the migration logic into workspaces.spec and simplifies the PR drastically.

@zmrow zmrow changed the title Add cargo make target that builds migrations Include migration builds in image builds Feb 7, 2020
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
Copy link
Member

@jahkeup jahkeup left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! There's a couple suggestions and questions inline. Approach seems sound to me otherwise!

packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
tools/rpm2img Outdated Show resolved Hide resolved
Makefile.toml Outdated Show resolved Hide resolved
@zmrow zmrow force-pushed the make-migrations branch 2 times, most recently from efac087 to 9eb641b Compare February 11, 2020 22:24
@zmrow
Copy link
Contributor Author

zmrow commented Feb 11, 2020

Reworked this quite a bit.

  • Split up the relevent portions of the Dockerfile. First we build the "repo", then we build the image with rpm2img, then we build the migrations with a new script rpm2migrations.
  • Build migrations as a package in workspaces.spec
  • Remove the test migration from the tree.

Dockerfile Show resolved Hide resolved
Dockerfile Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
packages/workspaces/workspaces.spec Outdated Show resolved Hide resolved
tools/rpm2migrations Outdated Show resolved Hide resolved
tools/rpm2migrations Outdated Show resolved Hide resolved
tools/rpm2migrations Outdated Show resolved Hide resolved
@zmrow
Copy link
Contributor Author

zmrow commented Feb 12, 2020

Addressed @bcressey 's comments

Copy link
Contributor

@tjkirch tjkirch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you please show the location/contents of the new artifact in the description?

Comment on lines +205 to +206
for version_path in %{_builddir}/workspaces/api/migration/migrations/*; do
for migration_path in "${version_path}"/*; do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we should be consistent about quoting. (I don't like quoting paths containing globs, but if shellcheck complains, it's fine.)

packages/workspaces/workspaces.spec Show resolved Hide resolved
tools/rpm2migrations Outdated Show resolved Hide resolved
tools/rpm2migrations Outdated Show resolved Hide resolved
This change includes migration builds with image builds. It
will build all of the migrations, lz4 compress them, and
tar them up as an output of the build.
@zmrow
Copy link
Contributor Author

zmrow commented Feb 13, 2020

Added contents of migrations artifact to description

Addressed @tjkirch 's comments

Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⛵️

@zmrow zmrow merged commit 1d3bc9a into develop Feb 14, 2020
@zmrow zmrow deleted the make-migrations branch February 14, 2020 16:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants