Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate Maven -> Gradle #1440

Closed
vlsi opened this issue Mar 14, 2019 · 112 comments
Closed

Migrate Maven -> Gradle #1440

vlsi opened this issue Mar 14, 2019 · 112 comments

Comments

@vlsi
Copy link
Member

vlsi commented Mar 14, 2019 •
edited
Loading

Gradle would simplify build process, and we would be able to avoid current pgjdbc-parent-poms dance.

Note to postgresql-jdbc-like package maintainers:

Some distributions miss Gradle, Kotlin, and other dependencies, so pgjdbc provides a convenience "source release artifact" which is a minimal Maven project to produce a similar to the official release jar artifact from "text files only".
The name of the source release archive in 42.2.15 will be https://repo1.maven.org/maven2/org/postgresql/postgresql/42.2.15/postgresql-42.2.15-jdbc-src.tar.gz.
The package should be buildable with mvn package.
@vlsi
Copy link
Member Author

vlsi commented Mar 14, 2019

@praiskup , do you know if Gradle is an option for Fedora packaging?

@praiskup
Copy link
Member

Unfortunately I don't know. Is there any package you know of that should be in fedora (some core stuff) which is using gradle? FWIW, from Fedora perspective I don't really understand what is the parent-poms for. :-)

@praiskup
Copy link
Member

praiskup commented Mar 14, 2019 •
edited
Loading

@kubco2, any idea? (FTR: Jakub is going to be the main package maintainer for postgresql-jdbc RPM package, I'll stay to help a bit as a co-maintainer).

@vlsi
Copy link
Member Author

vlsi commented Mar 14, 2019

Fedora perspective I don't really understand what is the parent-poms for. :-)

No-one knows :-)
It is there to workaround Maven and to factor out common build logic.

With Gradle all the code could be co-located in a single repository which would be easier to follow.

@praiskup
Copy link
Member

What would happen if parent-poms were in the pgjdbc/pgjdbc.git?

@vlsi
Copy link
Member Author

vlsi commented Mar 14, 2019

Technically speaking in Maven it has to be two distinct mvn calls.
In other words, we can't really make that in a single mvn project.

@praiskup
Copy link
Member

I suspect that we'll have to adapt somehow if upstream feels it's necessary; I have found other packages depending on maven-local.rpm so it should be somehow possible (but I have zero experience).

Note only that Fedora does offline builds, only against Fedora RPMs and nothing else. So any additional build dependency might hurt (and not only Fedora, I've seen that other distributions were able to adapt and package new pgjdbc versions as well).

@kubco2
Copy link

kubco2 commented Mar 18, 2019

I looked at the Gradle in Fedora and it is on the list of orphaned packages, it may be retired soon. I do not think it is good move in case of fedora packaging.

@davecramer
Copy link
Member

davecramer commented Mar 18, 2019 via email

@praiskup
Copy link
Member

I looked at the Gradle in Fedora and it is on the list of orphaned packages, it may be retired soon.

It rather seems that some people think that java should live in module only ... shrug. That said, doubt it will disappear from Fedora entirely, but it will live on a slightly different place.

@davecramer
Copy link
Member

there are ways to pull it in using sdkman when you build

@praiskup
Copy link
Member

I guess you mean bundling, which is possible in Fedora (though ugly) ... However, what is difficult in Fedora is a way more difficult in RHEL (moving to gradle would probably mean that we wont be able to get updated pgjdbc to RHEL 8).

@maffe maffe mentioned this issue Aug 29, 2019
Closed
2 tasks
@praiskup
Copy link
Member

I noticed this thread by accident (fedora is dropping gradle support):
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/BMJXGWKXXFOOBQON3XFYPFBOWEZMAKKU/#2OBMVCMHZEFEYRFG4XS3YFSF23ENGXNC

@davecramer
Copy link
Member

davecramer commented Sep 18, 2019 via email

@praiskup
Copy link
Member

So do you need to actually build gradle to be able to build packages ?

Probably (I haven't played with gradle so far, and I don't think I'll have to
because pgjdbc is not maintained by me in now in Fedora),
but usually if one needs something to build - he usually has to commit himself
for the maintenance (and java sig claims it is not possible in case of gradle).
Perhaps it is possible to reasonably bundle gradle, dunno.

Just pinging @odubaj (current Fedora maintainer), because @kubco2 left
the team.

@virtual-machinist
Copy link
Contributor

virtual-machinist commented Sep 19, 2019 •
edited
Loading

This will probably sound ignorant as I don't know anything about Fedora / CentOS / RHEL packaging, but can the problem be reasonably solved using Gradle Wrapper scripts?

You basically only need the correct JDK to be installed for the build. Wrapper can be instructed to "download" the full Gradle binary from a local file AFAIK, so all you need is the dependencies for the project, if any (haven't looked at the POM, too much XML 😛).

I rarely see a Bamboo or Jenkins CI server that has the latest (or any) Gradle version I would like to use when building applications, but the problem can be easily overcome with using wrapper.

@davecramer
Copy link
Member

davecramer commented Sep 20, 2019 via email

@vlsi
Copy link
Member Author

vlsi commented Sep 20, 2019

The whole thing is funny and not that funny at the same time.

For instance, JUnit 5 is built with Gradle: https://github.com/junit-team/junit5
At the built time they generate pom.xml files that list dependencies in Maven format:
https://repo1.maven.org/maven2/org/junit/jupiter/junit-jupiter-engine/5.5.2/junit-jupiter-engine-5.5.2.pom

Note: the pom files do not have entries that are related to the build steps. They contain just dependencies.

Then there's a Fedora package for junit5: https://src.fedoraproject.org/rpms/junit5/blob/master/f/junit5.spec
What they do is they download those pom files, and edit them up to a point where the code compiles with Maven.

I see two cases for a build system:

  1. A tool that developer uses to develop, test and release software. For instance, we want pgjdbc to be loadable to IDE, the dependency jars should be downloaded automatically and so on.
    Then we want to produce relevant jars for the supported Java versions, etc.

  2. "packagers" seem to focus that every package must be produced from the source code. In other words, they don't want to download Gradle from the official server. They would rather download Gradle sources, and build them.
    In the same way, they don't want to download "the official pgjdbc release", instead they want to build their own binary.

A bit of the fun is their compiler/environment/build process might have issues, so the produced binary might turn out to be ill.
For instance, Fedora's build of JUnit5 executes no unit tests for JUnit itself.
Would you trust using junit5.jar for running your unit tests in case you know that jar was not tested at all? The fact that it is built from some git source does not imply that the resulting jar works as expected.

Even though current Gradle builds from the source (of course it requires some Gradle version, so there's bootstrapping problem, however it is the same for GCC), it does pull a lot of dependencies. That means Fedora would have to build those dependencies from the source as well, and update when CVEs are discovered.

"Supporting Gradle package" might indeed sound non-trivial taking into the account that Gradle is not used a lot for package builds. However, here I just guess, and I don't know the true effort it requires to build Gradle in Fedora-like env.

So. What should we do about it?

I guess we do the following:

  1. We move to Gradle to simplify development/maintenance of pgjdbc
  2. Fedora packagers apply some regexps to build the jar (e.g. even with javac)

Note: I do not think there's a rule that the source code must always come from Git or other SCM. In other words, to simplify #2 we could publish a source release. That is the set of files (which might be already pre-processed).

Note: "keeping Git for developers only, and releasing software with source archives" is a well-established practice at the Apache Software Foundation.

I don't want to invest much into the support of two build systems at the same time (e.g. "Gradle for development" + "Ant/Maven for packagers"), however it won't probably take us much to produce a "source release" that is buildable with more-or-less trivial javac.

However, I would start with #1 and see what Fedora would do.

@praiskup
Copy link
Member

Well, I'd suggest to not move to gradle and stay on maven, and try to
avoid the parent-pom dance while staying with maven (if other projects
can?). Especially if that's the only motivation here.

There was a several-years window when distributions were not able to build
new version of pgjdbc, and it changed after the Fedora efforts... now the
pgjdbc module shipped in distros is mostly up2date). Testsuite can be run
just fine, and I was glad it's all in a very good shape. It would be
certainly regression from distro POV.

I'm not an expert on java topics; and unfortunately I'm not able to spend
too much time on pgjdbc packaging anymore. So I form this as a wish.

@davecramer
Copy link
Member

davecramer commented Sep 20, 2019 via email

@praiskup
Copy link
Member

In light of this who would be able to do packaging for pgjdbc if not you ?

@pkubatrh, @hhorak, @odubaj, @UncleAlbie and @ljavorsk.

@virtual-machinist
Copy link
Contributor

In other words, to simplify #2 we could publish a source release.

Isn't a source JAR basically a source release?

Note: the pom files do not have entries that are related to the build steps. They contain just dependencies.

From what I can see JUnit5 has a rather complex build. I wouldn't be surprised if they couldn't maintain both Maven and Gradle if they wanted to - "too much" build logic done in Gradle, some plugins may be either missing in Maven or have different API. So they basically applied maven-publish + a couple of bits to change the output POM. Same thing can in theory be done here, and I'm sure even test support can be mangled into the fake POM.

@vlsi
Copy link
Member Author

vlsi commented Sep 20, 2019

Isn't a source JAR basically a source release?

No, it is not. -source.jar is something that enables IDE to show the source code.
Typically it does not contain build scripts, resources (e.g. licenses for the third-party libraries that are embedded as a part of the release), and so on.

@hhorak
Copy link

hhorak commented Sep 23, 2019

@odubaj and @UncleAlbie should take care of pgjdbc in Fedora in the future, they've joined this "Java business" pretty recently, so this will be an interesting start for them.

For Gradle Wrapper, it was not mentioned here explicitly, but since that tool downloads the gradle if not present, it does not help at all in Fedora, we're simply required to build all packages from source in Fedora (no downloading possible during build).

@vlsi, a question for the idea with the "source release" -- do I understand correctly that the ant/maven build scripts/config would be maintained in parallel to the gradle build config?

@vlsi
Copy link
Member Author

vlsi commented Sep 23, 2019

a question for the idea with the "source release" -- do I understand correctly that the ant/maven build scripts/config would be maintained in parallel to the gradle build config?

I don't really know the complexity, however I don't want to maintain the two sets of the build scripts.
Of course, Fedora build doesn't really require advanced code verification stuff like "coverage reports, bytecode analysis, etc, etc", however things like META-INF/MANIFEST.MF creation might become non-trivial.

I see there might be "really crazy" move where we build pgjdbc.jar with Gradle, then we extract relevant META-INF/MANIFEST.MF from there and put it into "source release" which is compilable via trivial scripts as it has MANIFEST.MF already prepared.

It is not like "Gradle would make things 42 times simpler for pgjdbc", however Apache Maven does have limitations in terms of "making pgjdbc releases for multiple Java versions at the same time".

Typical source releases in Apache include the original build scripts.
Here you can find a sample: https://dist.apache.org/repos/dist/dev/jmeter/apache-jmeter-5.2.0-rc2/
apache-jmeter-5.2.0_src.zip is a "source release candidate" that is it contains the source code for Apache JMeter project, so everybody can build it.
It can be built with the included Gradle Wrapper (as you note, it would download binary from the Internet), or it can be built with pre-installed Gradle.

A site question would be "what if pgjdbc wants to add Kotlin as a dependency?"
Does that mean it can't be made as well?

@vlsi
Copy link
Member Author

vlsi commented Sep 23, 2019

Just in case, Bazel package is built in COPR out of 250MiB archive that contains all the jar dependencies included (see https://copr-dist-git.fedorainfracloud.org/cgit/vbatts/bazel/bazel.git/ )

It basically takes bazel-0.29.1-dist.zip from https://github.com/bazelbuild/bazel/releases and "builds" it.

What if we just bundle Gradle for the build purposes? (just kidding)

@vlsi
Copy link
Member Author

vlsi commented Mar 4, 2020

odubaj and UncleAlbie should take care of pgjdbc in Fedora in the future, they've joined this "Java business" pretty recently, so this will be an interesting start for them.

@odubaj, @UncleAlbie can you please have a look at Add source distribution commit here https://github.com/pgjdbc/pgjdbc/pull/1627/commits ?

So far copr build is working.
https://copr.fedorainfracloud.org/coprs/g/pgjdbc/pgjdbc-travis/build/1284538/

Can you please verify if the resulting artifacts work at all for Fedora's purposes, and can you please verify if .spec looks fine?

The idea is that upstream (pgjdbc) generates an archive like postgresql-42.3.0-src.tar.gz (the link is temporary)

postgresql-42.3.0-src.tar.gz contains the subset of the source files which is more-or-less enough to build the jar.

-src.tar.gz would differ from whatever is on GitHub (e.g. some files would have some variables with their replacement values)

-src.tar.gz would come with a PGP signature.

-src.tar.gz has a reduced set of tests.

There's an alternative way which might be something like is used for junit5

Frankly speaking, I love neither of those options.
In case with -src.tar.gz pgjdbc developers have to maintain the second build file (see reduced-pom.xml). In return, we can easily run mvn ... against that file and see if the jar compiles at all or not.

In junit5-way, pgjdbc developers lose the way to analyze the reasons for the failure.
For instance, I can't really reproduce %pom_xpath_set dance on my macOS :(

@odubaj
Copy link
Contributor

odubaj commented Mar 9, 2020

Can you please describe the option using reduced-pom.xml in more detail? Are there any other disadvantages than maintaining another pom.xml?

I understand that the package will use -src.tar.gz with reduced tests and other things, which I would say might not disturb us much (I am not expert in these technologies, there might be some issues I do not know), if compatibility to previous versions will be held.

In addintion, why is the reduced -src.tar.gz available on new url (https://repo1.maven.org/maven2/org/postgresql/postgresql/%{version}/postgresql-%{version}-src.tar.gz) ?
Are there any changes to this github repository, which we cannot see here ?

From my point of view, the option where developer or maintainer cannot debug or analyze some problems is dangerous. If problems occur, it can be very disturbing for boths sides. We can talk more about this option, but the package must be debbugable by both sides...

@vlsi
Copy link
Member Author

vlsi commented Mar 9, 2020

with reduced tests and other things, which I would say might not disturb us much

Technically speaking, the binary will be different, and it might easily fail to work.

In addintion, why is the reduced -src.tar.gz available on new url

-src.tar.gz is a completely new file, and it was never provided by pgjdbc.
I don't quite get what do you mean by available on new url

Are there any other disadvantages than maintaining another pom.xml?

E.g. Gradle-based build is reproducible, while Maven-based is not.

@davecramer
Copy link
Member

davecramer commented Aug 10, 2020 via email

@df7cb
Copy link
Contributor

df7cb commented Aug 10, 2020

Sorry about the last comment (now deleted), I didn't realize I have two checkouts of pgjdbc on this machine and used the wrong one. Scram does get bundled correctly now. (There are still some problems with building on older Debians but that's a different issue I think.)

@davecramer
Copy link
Member

Thanks for the clarification!

@df7cb
Copy link
Contributor

df7cb commented Aug 10, 2020

@davecramer: The error was that the query shown didn't get executed and it printed some internal "StringPreparation" instead. The test script used is https://salsa.debian.org/java-team/libpostgresql-jdbc-java/-/blob/master/debian/tests/sqlline

@vlsi
Copy link
Member Author

vlsi commented Aug 10, 2020

@df7cb , can you please clarify if the error still appears if you use the latest pgjdbc?

@df7cb
Copy link
Contributor

df7cb commented Aug 10, 2020

@vlsi: Everything works fine now with the 42.2.15 tarball. Thanks!

@df7cb df7cb mentioned this issue Aug 11, 2020
Closed
@devrimgunduz
Copy link

Hi,

Can you please fix the tarball, and put everthing under the root directory? Currently it is:

pgjdbc-REL42.2.15/postgresql-42.2.15-jdbc-src/

Also, can you please fix the tarball name? It used to be:

postgresql-jdbc-42.2.14.src.tar.gz

but now it is:
postgresql-42.2.15-jdbc-src.tar.gz

Thanks!

@devrimgunduz
Copy link

Also, packages are lagging way behind. Any chance you can release .15 soon?

@vlsi
Copy link
Member Author

vlsi commented Aug 12, 2020 •
edited
Loading

Can you please fix the tarball, and put everthing under the root directory? Currently it is:

tar xzf postgresql-42.2.11-dist.tar.gz produces postgresql-42.2.15-jdbc-src/ folder with everything inside.
I do not think the archive contains pgjdbc-REL42.2.15/ folder.

$ LC_ALL=en_US.utf-8 tar -tvf postgresql-42.2.15-jdbc-src.tar.gz | head -5
drwxr-xr-x  0 0      0           0 Jan  2  1970 postgresql-42.2.15-jdbc-src/
-rw-rw-r--  0 0      0        1321 Jan  2  1970 postgresql-42.2.15-jdbc-src/LICENSE
-rw-rw-r--  0 0      0       14115 Jan  2  1970 postgresql-42.2.15-jdbc-src/README.md
-rw-rw-r--  0 0      0         511 Jan  2  1970 postgresql-42.2.15-jdbc-src/build.properties
-rw-rw-r--  0 0      0          39 Jan  2  1970 postgresql-42.2.15-jdbc-src/ssltest.properties

Do you suggest we should drop postgresql-42.2.15-jdbc-src/ folder?
I guess it is common practice to keep that, so the users don't pollute their filesystem by accidental extraction to their home or whatever directory. I guess tar --strip-components 1 could help to strip the root directory if it is unwanted.

Also, can you please fix the tarball name? It used to be:
postgresql-jdbc-42.2.14.src.tar.gz

AFAIK the previous name was postgresql-42.2.14-src.tar.gz (see #1440 (comment) ), and somebody noticed it was extremely confusing as the name looks like the sources for PostgreSQL database itself.

That is why we added -jdbc. However, we follow Maven artifact conventions which are ${artifact.name}-${version}${classifier}${extension}.
In other words, every file in https://oss.sonatype.org/content/repositories/orgpostgresql-1240/org/postgresql/postgresql/42.2.15/ folder must start with postgresql-42.2.15.
Currently, we use -jdbc classifier, which enables us to upload -jdbc.src.tar.gz side by side with the .jar file (see https://oss.sonatype.org/content/repositories/orgpostgresql-1240/org/postgresql/postgresql/42.2.15/ )

We do not really want to use a new artifact name for ...src.tar.gz as it would require us to create an artificial pom.xml file (for instance it would be https://oss.sonatype.org/content/repositories/orgpostgresql-1240/org/postgresql/postgresql-jdbc/42.2.15/ directory which would be extremely confusing for Java consumers as they won't be able to tell if they should use postgresql or postgresql-jdbc)

@vlsi
Copy link
Member Author

vlsi commented Aug 13, 2020

Also, packages are lagging way behind. Any chance you can release .15 soon?

@devrimgunduz , @df7cb we can release 42.2.15 as soon as you confirm postgresql-42.2.15-jdbc-src.tar.gz works for you.

@df7cb
Copy link
Contributor

df7cb commented Aug 13, 2020

As said above, 4.2.15 works for me. Thanks!

@devrimgunduz
Copy link

I still don't like the tarball name. I respect Maven requirements, but I can also please get
postgresql-jdbc-42.2.15.src.tar.gz

along with the others? Thanks.

@df7cb
Copy link
Contributor

df7cb commented Aug 13, 2020

(Debian renames the tarballs anyway to ...orig.tar.gz so I don't have any preference here.)

@vlsi
Copy link
Member Author

vlsi commented Aug 13, 2020

To my best knowledge, I can't make Maven file to be like postgresql-jdbc-...
At least it is not something we can incorporate in a timely manner.

So the only option left is to publish postgresql-jdbc-... somewhere else as a part of the release.
It might be https://jdbc.postgresql.org/download.html, however, jdbc.postgresql.org is updated manually, so it would incur overhead to @davecramer , and it would make testing more complicated.

The current postgresql-42.2.15-jdbc-src.tar.gz file is published automatically, and it takes me zero effort to prepare/upload/remove a release candidate. If you stick with jdbc.postgresql.org/.../postgresql-jdbc..., then you would lose the ability to test release candidates.

Just in case if you missed: if we release https://oss.sonatype.org/content/repositories/orgpostgresql-1240/org/postgresql/postgresql/42.2.15/ , then the release file would be byte-by-byte the same thing as you just saw.

@devrimgunduz
Copy link

Testing what? We are talking about the same tarball, with just a different name.

"postgresql"-version is a broken naming, sorry.

@vlsi
Copy link
Member Author

vlsi commented Aug 13, 2020

Testing what?

Testing release candidates.

"postgresql"-version is a broken naming, sorry.

Don't get me wrong. I think it is wrong naming event for Java clients, as they have something like postgresql-$version.jar.
However, internet is full of references to org.postgresql:postgresql:$version, so migration to a new name is not that easy.

@odubaj
Copy link
Contributor

odubaj commented Aug 13, 2020

Hi, due to missing gradle in Fedora, I am forced to use maven for building this project. I am also experiencing some dependency problems. For example missing "Checker Framework" and many other (hopefully solvable) problems. Any suggestions, how to resolve at least the problem with checker framework ?
Thanks.

@vlsi
Copy link
Member Author

vlsi commented Aug 13, 2020

@odubaj , please check #1440 (comment). The current release candidate is https://oss.sonatype.org/content/repositories/orgpostgresql-1240/org/postgresql/postgresql/42.2.15/postgresql-42.2.15-jdbc-src.tar.gz

@df7cb
Copy link
Contributor

df7cb commented Aug 14, 2020

Hi, due to missing gradle in Fedora, I am forced to use maven for building this project. I am also experiencing some dependency problems. For example missing "Checker Framework" and many other (hopefully solvable) problems. Any suggestions, how to resolve at least the problem with checker framework ?

Not sure if it answers this question, but I did have to remove some test deps from maven's pom.xml for Debian, and the build still works: https://salsa.debian.org/java-team/libpostgresql-jdbc-java/-/blob/master/debian/patches/missing-test-deps

@odubaj
Copy link
Contributor

odubaj commented Aug 17, 2020

No changes, still missing dependencies of org.checkerframework.*

Build here:
https://koji.fedoraproject.org/koji/taskinfo?taskID=49428009

@vlsi
Copy link
Member Author

vlsi commented Aug 17, 2020

@odubaj , can you please clarify which ..spec files you are using for building the driver?
Do you use driver-maintained https://github.com/pgjdbc/pgjdbc/blob/0aa6d2e8d3e4f2a2577cae680dc425289c54e535/packaging/rpm/postgresql-jdbc.spec.tpl or do you use your own one?

@odubaj
Copy link
Contributor

odubaj commented Aug 17, 2020

I use my own one, I will also use the driver maintained and see what will happen

@odubaj
Copy link
Contributor

odubaj commented Aug 17, 2020

My custom .spec is very similar to the driver maintained one. I am afraid I won't be able to build this package without explicitly bundling the missing dependency (not a good option) or somehow disabling the org.checkerframework.* tools.

@vlsi
Copy link
Member Author

vlsi commented Aug 17, 2020

@odubaj , are you sure you use https://repo1.maven.org/maven2/org/postgresql/postgresql/%{version}/postgresql-%{version}-jdbc-src.tar.gz link to get the sources?

I'm afraid you are not using it, which causes the error.
Note: postgresql-%{version}-jdbc-src.tar.gz has all the checkerframework uses commented-out, so that build does not require the dependency at all.

@odubaj
Copy link
Contributor

odubaj commented Aug 17, 2020

Sorry my mistake, we were using the old URL pointing to tags in this repository.

@vlsi
Copy link
Member Author

vlsi commented Aug 17, 2020

On top of that, I guess you are still using pgjdbc-parent-poms, which is no longer needed. postgresql-%{version}-jdbc-src.tar.gz should be enough.

I'm closing the issue, as migration to Gradle has already been made, and source release artifacts are also provided.

@vlsi vlsi closed this as completed Aug 17, 2020
@odubaj
Copy link
Contributor

odubaj commented Aug 17, 2020

Thanks for help!

@zmiklank zmiklank mentioned this issue Jan 20, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@ebourg @vlsi @devrimgunduz @davecramer @df7cb @hhorak @kubco2 @virtual-machinist @praiskup @odubaj