New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Ubuntu Build in Pipeline #6030
Add Ubuntu Build in Pipeline #6030
Conversation
There is a lot to unpack in this PR:
It seems like the first two are not strict requirements for the last one and they significantly contribute to the "visual" clutter in reviewing the new packaging tasks. I think it makes sense to split this into three separate PRs. At this point, I'm not sold on switching from |
+1 to this idea. Losing input validation outright does not seem desirable. The toolsmiths are actively working on upgrades so it won't be long until the features of v4.1.0 are available. In the interim, if there are specific variable interpolations that do require
+1 to this as well. I'd like to hold off adding review comments to the PR if everyone agrees that it should be split up. |
74d0759
to
8a22df9
Compare
@kmacoskey @bradfordboyle Good point about the {{}} change. We removed that commit. (Turns out we're not using the secrets file that has nested keys after all, so we don't need to do that at this time.) Regarding splitting this into multiple PRs: Part of the motivation for this is now gone with the removal of the {{}} change. Further, the commit for the So there are 2 main changes in this PR: the rename, and the "meat" of this PR, all the rest of the commits. What do you think of reviewing all the commits separately (see "Commits" tab) within one PR? If not, we can separate out into 2 PRs if it hard to make sense of this; feeling like the big issue was the {{}} change... |
d70fa8c
to
7d983c9
Compare
Lets pause on this PR until we can have some context sharing about this from a release engineering perspective. Comments in the PR may not be the best place to start this conversation. |
@jpatel-pivotal Thank you for providing some insight into the longer term plans! Getting this PR into 5X_STABLE is still the goal and next step. |
bucket: ((bucket-name)) | ||
region_name: ((aws-region)) | ||
secret_access_key: ((bucket-secret-access-key)) | ||
regexp: deb_package_ubuntu16/greenplum-db_(.*)_amd64.deb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please correct me if I am mistaken, but the regexp here resolves to VERSION=$(</tmp/gpdb_src/VERSION awk '{print $1}')
?
Is there a downstream reason it's preferable to have the SHA within the Debian package name in s3? The other blobs being exported to ((bucket-name))
are all generic names like bin_gpdb.tar.gz
so that downstream processes can consume those versioned objects (it's a versioned bucket) as versioned_file
s. With a SHA in the name you cannot consume either as a regexp
, because it's not semi-semantically sortable, or as a versioned_filed
because each object name will be unique.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, this is using the regexp to find the version information, and we follow the extended semantic versioning with +
to additional information. We need the SHA in the file name to ensure the downstream packaging knows exactly which commit we depend on.
It's way easier to help us check which feature in and which feature out by containing a commit SHA here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Which part of the downstream packaging needs to know this SHA? Typically any downstream packaging should be based on either using release candidates, which means just consuming the latest versioned object out of an s3 bucket, or be based on an actual release which is driven by a tag on a repository.
What specifically drives knowing exactly which SHA should be consumed in any downstream process?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kmacoskey regarding the bucket-file naming with included SHA, the key consumer of that file in deliverables/ will be code refactored from the gp-enterprise-debian-package repo. Currently, the SHA does not appear to be used. In particular, the version discovery in Makefile discovers the Greenplum version by running greenplum. @xinzweb may have more context.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kmacoskey For official release, we don't need the SHA, since the release tag of the GPDB is good enough. If we are doing the bleeding edge testing to verify the certain behavior of GPDB (not released yet) in our environment, then we need to know which SHA we are referring to.
What's the additional complexity you worried about having the .deb file with the version and SHA? Maybe we overlooked the naming issue here. Thanks.
@@ -1019,6 +1053,8 @@ jobs: | |||
- get: centos-mingw | |||
- get: ubuntu-gpdb-dev-16 | |||
- get: ubuntu-gpdb-debian-dev-16 | |||
- get: docker-in-concourse |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
While it does fit the pattern, it's not necessary to have this resource, docker-in-concourse
, pass through the gate_compile_start
job. I believe it would be reasonable to completely remove that job at this point. It's original purpose for staggered starts and prior to the pipeline template generator has long since past.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point. I was wondering on the same thing, but just want to follow the pattern. We will remove it.
- get: gpaddon_src | ||
passed: [gate_compile_start] | ||
- get: docker-in-concourse | ||
passed: [gate_compile_start] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Related to the above, this resource doesn't need to pass through gate_compile_start
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's great to know that. That should be easy to fix.
Is there a dev copy of this pipeline set anywhere? I'd like to see that the pipeline is green and to take a look at the build output for the new job. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This PR seems to follow a consistent pattern of inline shell scripts in task yaml and Dockerfile. What does this pattern do better than having this logic as a separate bash script? I would think that separate scripts would increase the potential reuse and make it easier to test/lint.
Other than that, this looks OK.
args: | ||
- -ec | ||
- | | ||
. docker-in-concourse/dind.bash |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems like a lot of bash in a task definition. Is there a reason to not have this as a bash script?
WORKDIR /tmp/gpdb_src | ||
|
||
RUN \ | ||
VERSION=$(</tmp/gpdb_src/VERSION awk '{print $1}') && \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a reason to not have this be a separate script file and then COPY
and RUN
in the dockerfile?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given this job being run in a concourse pipeline, what is this error that occurs a large number of times? A single build included this error in the stdout 966
times.
ERROR: ld.so: object 'libfakeroot-sysv.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
When the deb package is built at the end of the job execution there is another echoing of the contents of a built gpdb binary directory. Could this be avoided? It's adding to the already large amount of stdout generated by this concourse job and I am unsure of the value of having this particular output included in the stdout.
Example output:
E: greenplum-db: dir-or-file-in-opt opt/gpdb/bin/.rcfile
E: greenplum-db: dir-or-file-in-opt opt/gpdb/bin/README
E: greenplum-db: dir-or-file-in-opt opt/gpdb/bin/analyzedb
@@ -0,0 +1,95 @@ | |||
FROM pivotaldata/ubuntu-gpdb-dev:16.04_gcc_6_4 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We are currently only using GCC 6.2 to compile gpdb5. Has switching to 6.4 been validated as an ok change? I'd suggest just sticking with 6.2 and let compiler changes be driven by more comprehensive testing, as well it could be confusing for support if there are ever issues down that road that have been introduced by using 6.4 but it is not known that this particular platform is compiled with 6.4 while the rest of the platforms are with 6.2.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, please see email to directors to get their opinion on this.
ec39cde
to
2143cd5
Compare
@bradfordboyle @kmacoskey We added 4 commits for the refactors you suggested, extracting shell files out of the concourse yaml, and quieting both Regarding the final issue Kris mentioned: I recall discussing this during LL; I recall we formerly pinned the Thanks, Larry & Fei |
4701031
to
a513e91
Compare
I don't really have a strong opinion on this PR given what it is doing and the fact that it is doing it on 5. On Master I think that it is important for all of the Compile, testing packaging etc. jobs to be the same. Meaning that changes made to one should be made to all. |
0dfdd44
to
e09a2fd
Compare
I took a look at this, and I see two things that I am trying to wrap my head around:
|
@doty-pivotal I will reach out and we can talk through the topics in your comment this am. Thanks |
15ffab7
to
4f75212
Compare
4f75212
to
bdfd919
Compare
4dbda90
to
fc7e8d7
Compare
6866389
to
bb47677
Compare
@kmacoskey we have cleaned up and squashed them down to a simpler commit history. Let us know as we would like to merge this PR in sooner rather than later. |
[#160667453] Authored-by: David Sharp <dsharp@pivotal.io>
Co-authored-by: Xin Zhang <xzhang@pivotal.io> Co-authored-by: David Sharp <dsharp@pivotal.io> Co-authored-by: Larry Hamel <lhamel@pivotal.io>
bb47677
to
637f01a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks as though the generated pipeline based on the changes to gpdb-tpl.yml
is not included in the PR. If you generate the pipeline based on the template, there are a number of issues that will need to be resolved.
…ntu16 Authored-by: David Sharp <dsharp@pivotal.io>
Authored-by: David Sharp <dsharp@pivotal.io>
- Copy out the gpdb_bin.tar.gz at the end of build - for details, please see /src/tools/docker/ubuntu16/README.md - Skip building gphdfs, orafce, pgbench, pgbouncer on Ubuntu [#160667453] Co-authored-by: Xin Zhang <xzhang@pivotal.io> Co-authored-by: Fei Yang <fyang@pivotal.io> Co-authored-by: David Sharp <dsharp@pivotal.io> Co-authored-by: Goutam Tadi <gtadi@pivotal.io> Co-authored-by: Larry Hamel <lhamel@pivotal.io> Co-authored-by: Jemish Patel <jpatel@pivotal.io>
1e00000
to
c6cf8ff
Compare
@kmacoskey should be all good now. PTAL. |
Ok, in order to confirm things on my side i've checked that the template matches the |
Co-authored-by: Xin Zhang <xzhang@pivotal.io> Co-authored-by: Fei Yang <fyang@pivotal.io> Co-authored-by: Larry Hamel <lhamel@pivotal.io> Co-authored-by: David Sharp <dsharp@pivotal.io> Co-authored-by: Goutam Tadi <gtadi@pivotal.io>
Co-authored-by: Xin Zhang <xzhang@pivotal.io> Co-authored-by: Larry Hamel <lhamel@pivotal.io> Co-authored-by: David Sharp <dsharp@pivotal.io> Co-authored-by: Fei Yang <fyang@pivotal.io>
c6cf8ff
to
559c7ae
Compare
ubuntu-16.04/
toubuntu16-runner/