New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replacing the current build system (Ant) with Gradle #751

Open
Nava2 opened this Issue Feb 22, 2016 · 137 comments

Comments

@Nava2
Member

Nava2 commented Feb 22, 2016

Brief Description

Currently, we use Apache Ant which does the job as we need it. Unfortunately, as the project grows and becomes more complex, we are adding complexity and require flows that don't "blend" with Ant's XML build files (e.g. 1, 2, 3). I fear that over time, this system has developed a significant "code-smell" that will degrade over time.

Proposal

I would like to use a more modern build system, Gradle. Gradle is a Groovy-based build tool that allows the full expressiveness of Groovy with the build power of Ant/Maven.

To do this, I would like to do this in parallel with the Ant build. I believe it is "too big too fast" to swap it out without running both side-by-side for a period. Luckily, our current CI tools will allow us to do this as time progresses. Additionally, there would need to be milestones before eventually deprecating the old system (i.e. updating documentation, getting individuals up to speed). I would like to keep simple tasks available, for example: gradle quickTest:BuilderTest would run the cheatsheet equivalent of: ant -DshowJunitOutput=true -Djunit=BuilderTest -f build.umple.xml resetUmpleSelf compile template.test.

This issue is a discussion thread, there would need to be more milestones created and "attainable goals" should this be accepted.

Pros

  1. Full access to Groovy
  2. Build logic works in dependency graphs (only build a->b->c if necessary)
  3. Battle tested by massive projects (e.g. Android)
  4. Allows for polygot builds
  5. Automatic parallelism in the build chain based on dependency graphs
  6. Tight integration with Ivy
  7. Acceptable point to improve/change build commands

Cons

  1. New tooling, new work flows
  2. Will require time running in parallel
  3. Groovy is not XML (which everyone knows)
  4. Change

Counter build tools

Maven

Maven is another XML build tool, but due to the XML nature of it, I do not see it being a significant benefit over the current Ant implementation. I also have a personal bias against it for being overly complicated.

@Nava2 Nava2 added the build label Feb 22, 2016

@Nava2 Nava2 changed the title from Proposal: Replacing the current build systel to Proposal: Replacing the current build system Feb 22, 2016

@Nava2 Nava2 added the proposal label Feb 22, 2016

@vahdat-ab

This comment has been minimized.

Member

vahdat-ab commented Feb 22, 2016

I'm also supporting the idea :-).
It's a modern build tool that can improve and simplify our build process. I've tried once before to do some small build parts with Gradle, it was fun and easy especially with the plugin architecture (I stopped because didn't have time to spend on it).
Furthermore, the new generation of students is familiar with Gradle and Maven so the involvement in the future of Umple will be easier and substantial.

For sure, the process must be done in parallel with current build process so it can be tested continuously. When we are done with documentation, we can switch to it.

Since admins may or may not be familiar with Gradle and would like to learn the process, I would suggest to commit small parts and let other active members read the code and understand the whole process. This helps to have expert members in Gradle over time.

I would also like to participate in the development process of new build sytsem.

@TimLethbridge

This comment has been minimized.

Member

TimLethbridge commented Feb 22, 2016

Yes, I agree with this. It seems to be the way of the future.

@TimLethbridge TimLethbridge changed the title from Proposal: Replacing the current build system to Proposal: Replacing the current build system with Gradle Feb 23, 2016

@ahochheiden ahochheiden self-assigned this Oct 4, 2016

@ahochheiden ahochheiden changed the title from Proposal: Replacing the current build system with Gradle to Replacing the current build system (Ant) with Gradle Oct 5, 2016

@ahochheiden ahochheiden removed the proposal label Oct 5, 2016

@ahochheiden

This comment has been minimized.

Contributor

ahochheiden commented Oct 5, 2016

I have chosen this issue as my term project. I will work with Vahdat and Tim to incrementally add portions of our build to Gradle.

As mentioned by Vahdat, the goal is that both Ant and Gradle build systems will work at the same time. Once we're confident in the Gradle build, we will work towards phasing out Ant.

@ahochheiden

This comment has been minimized.

Contributor

ahochheiden commented Oct 6, 2016

I read through a lot of the Gradle documentation. For our incremental approach, I think the right place to start would be to just import our Ant build into Gradle. (https://docs.gradle.org/current/userguide/ant.html#sec:import_ant_build) From what I understand, this would just wrap our current Ant build, so we could invoke it with Gradle commands and take advantage of some Gradle features.

I'm in the process of getting this to work. The import worked fine, and I can run all our Ant scripts through Gradle, the problem is that they're all failing.

C:\Users\Alex\Documents\GitHub\umple>gradle build
[ant:echo] Using myenv=wlocal
[ant:echo] Last Version: ${last.version}
:build
[ant:taskdef] Could not load definitions from resource org/apache/ivy/ant/antlib.xml. It could not be found.
[ant:taskdef] Could not load definitions from resource net/sf/antcontrib/antcontrib.properties. It could not be found.
:build FAILED

The problem seems to be related to Ivy. I remember that during the Dev Environment setup I did not install Ivy throughChocolatey, it was through Eclipse. I went and installed Ivy through Chocolatey, but that did not resolve this issue.

I'm going to look through our existing Ant scripts to see if I can find the source of these paths, or if they're being added by Gradle instead. If Gradle is adding them, the solution will probably be to add the paths myself. I'll probably ask Vahdat some questions about Ivy in private.

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 6, 2016

First off, I'm still lurking these repositories, and I'm sorry I couldn't help directly.

With this issue, I would really really like you to stay away from implementing this in "terms" of Ant. The two builds should run simultaneously side-by-side, but do not "share" between them. The reason for this is that Gradle and Ant follow very different design philosophies and if you mix them you may create inter-dependencies that could never be removed.

For example, with your Ivy problem, Ivy isn't needed with Gradle. You just specify dependencies within the dependencies block and you're done. You will be able to literally copy the "parameters" for each dependency and replace it within these dependency blocks (link).

I believe this issue would also involve changing the layout of the project tree. Because the build scripts are truly just Groovy scripts, we can do a lot more powerful (and less "ant" obnoxious) things with the build chain. For example, checking internet connectivity is much more straight-forward.

Changing the build process creates a lot of opportunity for changing things like project setup (including repository setup (@aforward)). Think about what the "end goal" would be for this outcome and try to design the idea "modern" version of this build tree. Ultimately, because of the flexibility with Gradle, we can "propose future changes," get them working with Gradle and Ant. Replace the Ant build after its working side-by-side and equivalently, then make changes that Ant wouldn't like.

If @vahdat-ab would allow it, I'd happily help with mentoring on this issue. I wanted to do it for a long time, but I've become so busy with my own projects I let this fall to the wayside. Ultimately, you're welcome to ask me questions regardless. 👍 I've actually picked up some Gradle knowledge since opening this months ago.

@aforward

This comment has been minimized.

Contributor

aforward commented Oct 6, 2016

I concur.

If you implement the "new" in terms of the "old" and the implementation level, then I would argue "why bother". There must be a real pain point that we are addresses (or possibly an "opportunity" point).

That said a few extra points....

  1. The work must be done in small increments; please no big bang as the build system is crucial to developers remaining agile about making changes and mitigating regressions

  2. Both systems (Ant vs Gradle) should achieve the same high level goals, but as @Nava2 points out don't shoot yourself in the foot by forcing a dependency between the new and old.

Good luck!

@TimLethbridge

This comment has been minimized.

Member

TimLethbridge commented Oct 6, 2016

I would be super-happy to have Kevin (aka @Nava2 ) mentor this work.

@vahdat-ab

This comment has been minimized.

Member

vahdat-ab commented Oct 6, 2016

@Nava2 There is no issue with that. We are all here to help and work with each other. In fact, this is a great thing we can expect from people who are working in Umple team.
I completely agree with you guys and I have already pointed out this in the response to @ahochheiden.
The reason behind this might come from the fact that reading the ant scripts looks hard and refactoring that makes the process much easier.
However, as @Nava2 said we might end up with some dependencies that are not going to be fixed later.
I completely agree to develop from scratch. It might takes time but it's the best approach.

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 7, 2016

@vahdat-ab, and @ahochheiden, why don't we setup some time early next week to chat about this? I'll have some more time then.

From @aforward:

  1. The work must be done in small increments; please no big bang as the build system is crucial to developers remaining agile about making changes and mitigating regressions

Nice thing about Gradle, since it's still Groovy, we can have methods that we change the implementation from down the road. Code for big changes, but implement them later (thinking along the lines of separate repository stuff).

@vahdat-ab

This comment has been minimized.

Member

vahdat-ab commented Oct 9, 2016

@Nava2 @ahochheiden I agree with that. I'm fully busy Tuesday but we can make it on Wednesday or Thursday. Of course, if you guys are ok with it as well.
I think it will be great to have @TimLethbridge

@ahochheiden

This comment has been minimized.

Contributor

ahochheiden commented Oct 9, 2016

Tuesday works for me, I'm free between 1:00 PM and 4:00 PM MST. Thursday would have to be before 1:00 PM MST or after 4:30 PM MST.

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 9, 2016

Wednesday before 430PM EST and Thursday are open for me this week. What about 1PM EST (11AM MST) on Thursday?

@ahochheiden

This comment has been minimized.

Contributor

ahochheiden commented Oct 12, 2016

1PM EST on Thursday it is. Do I need to add you on Skype or will Vahdat connect us?

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 12, 2016

Hangouts is easier for me.

@vahdat-ab

This comment has been minimized.

Member

vahdat-ab commented Oct 12, 2016

I'm fine with both. @Nava2 can you make it on skype? otherwise, we will switch to Hangouts.

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 12, 2016

Yup, kevin_brightwell, I think.

@TimLethbridge

This comment has been minimized.

Member

TimLethbridge commented Oct 12, 2016

Thanks so much for this. I am 'half way' back from my trip, but will be in the air for this, so I leave it to you. Thanks.

@Nava2

This comment has been minimized.

Member

Nava2 commented Oct 13, 2016

From our skype call:

Next steps

Create a plan for the milestones listed. Estimate some times, but don't feel too bad about hitting later ones. We are all in new territory with regards to how long this stuff will take (especially since you're new to Groovy!). If you're comfortable with it, I suggest using Githubs project management tools to setup the milestones better than this list here.

https://guides.github.com/features/issues/#filtering

Milestones

  • Create a hello world plugin
  • Have the plugin load the Umple JAR
    • Use a hard-coded path for now
  • Create simple tests for your simple plugin 😄
  • Call the compiling interface for the source files to convert to Java
  • Enhance the plugin to:
    • allow for specifying other generators (e.g. PHP or Ruby)
    • allow a specification for Umple jar versions or a local JAR path
    • figure out how to specify multiple generations for a single source tree in a build
      • We need this for running the Test Bed

Throughout these milestones, create a "paper trail" of what you're doing. If you run into problems, write it down. When you hit milestones, try to convert the current paper trail into a Wiki document. Keep the Wiki pages in the main Wiki, but as @vahdat-ab suggested in the call, use the other repository for issues.

Additionally, like the umple project, everything should be done in TDD format, make tests for everything as you do it, Red-Green-Refactor!

@vahdat-ab

This comment has been minimized.

Member

vahdat-ab commented Feb 24, 2017

I think is valuable to keep "standard" with. The "gradle/maven-way" is to setup sources like: src/main/umple, src/test/umple, src/main/java, etc.

As long as that structure is under the folder 'cruise.umple`, there is no problem. The names are not important. The reason I want to keep all thing inside the cruise.umple is if someone wants to setup the project in an IDE, there is not need to deal with many dependecies.

The only simplification that I can see is not depending on building sources. We will need to find a mechanism to "hot-swap" the umple classes in the build script if a "modern" version is available such that the plugin can use that over the "current stable." This task will need to be addressed, but using a single root project doesn't solve this in any way.

If it's possible to compile subprojects separately and build the final Umple compiler from them, there is no issue. I think this should be done with incremental compilation in Gradle.

Kevin clarifies a point that I didn't do a good job of explaining. We're not actually doing the compilation of Java files in the Umple plugin. In the Umple plugin, we're just telling the Java plugin where the generated Java files are (that is, we're telling it where the generated files from the generatedSource source set are located). We'll still only be doing the compilation once when we run the "create umple jar" task in the subproject.

That was my point as well. No need for our plugin to do it.

My only question is how we're going to resolve dependencies between subprojects. For example, as Vahdat mentioned, if we go to compile cruise.umple by itself,

Keep an eye on this :-)

regarding source sets, we should not mix two different concepts. We need to have our own source sets for Umple files. We shouldn't get any data from java source set for compiling umple files. This makes a dependency which is not required.

Finally, I have a question @AdamBJ : what is stopping you from implementing the first phase which is about code generation of projects and subprojects?

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Feb 24, 2017

I had a look at the Xtext link. It seems like we access the Xtext properties we add is via the extension object attached to the project that's passed into the plugin. That doesn't seem much better than packaging all our properties into a "Umple properties" object that we attach to the project extension object. There's less boilerplate if we create or own object, not to mention that if we create or own properties object we can name it something more intuitive than "Xtext".

I'm in favour of creating our own object and moving on to another issue. If we have time we can try to come up with a better solution.

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Feb 24, 2017

Vahdat, I've implemented a prototype that has a report project and subprojects. It's working (java is being generated from Umple files and compiled) and I can start working on the "real build" anytime. What's stopping me from doing that is two things.

First, we need to figure out to do about passing configuration information to the plugin. I've stated my preferences in my last post. Second, we need integration tests for the plugin. I've got a few already setup and working, but there are a few more I want to implement.

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Feb 25, 2017

@Nava2 (and Vahdat), to get going with the locally built jar change the buildscript closure in the build.gradle file of the project you want to apply the plugin to to something like this:

buildscript {
  repositories {
    maven {
      url uri('libs')
    }
     flatDir dirs: "../umple.gradle/build/libs"
  }
  dependencies {
    classpath files('libs/umple-latest.jar')
    classpath "umple.gradle.plugin:UmpleGradlePlugin-0.1.1"
  }
}

You'll need to have flatDir dirs to point to the location of your locally-build umple plugin jar

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Feb 26, 2017

I'm trying to update the plugin on the portal, but I don't have write permissions. Do either of you know how I can get write access?

@Nava2

This comment has been minimized.

Member

Nava2 commented Feb 26, 2017

I'm trying to update the plugin on the portal, but I don't have write permissions. Do either of you know how I can get write access?

I think @ahochheiden or @TimLethbridge will have access or know. The credentials should probably be held by Tim.

@Nava2

This comment has been minimized.

Member

Nava2 commented Feb 26, 2017

Additionally, at this point, this issue is getting cluttered. I think we should break down the next tasks and start working on them in a separate thread -- saving this one for updates. Keep all of the direct conversation in the other issue tracker. There's a lot of comments in this thread.

@TimLethbridge

This comment has been minimized.

Member

TimLethbridge commented Feb 28, 2017

I have the credentials to log in to the gradle plugin hub. It seems you have managed to deal with this as I see new updates in the last couple of days.

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Feb 28, 2017

@TimLethbridge Thanks for checking, I got them from Vahdat

@TimLethbridge

This comment has been minimized.

Member

TimLethbridge commented Nov 21, 2017

We have been working on this issue for ages. If it is to be continued we will need a really talented student who can drive it through to completion. The ant builds are complex and continue to evolve meanwhile ...

@AdamBJ

This comment has been minimized.

Contributor

AdamBJ commented Nov 21, 2017

@TimLethbridge @Nava2 When I finished my UCOSP term working on this (Spring 2017), the Umple Gradle plugin was 90-95% done, and I put together a "proof of concept" build that builds the Umple compiler jar using the Gradle plugin..

My hope is that the next student that comes along can use the Umple Gradle plugin as a "black box", and, using the PoC build as a starting point, begin work on converting the build to Gradle fully. Before that happens, ideally Kevin and I could check off a few (mostly testing related) boxes related to the Umple Gradle plugin. There are a couple open PRs from some work I did over the summer, but I haven't had a chance to respond to Kevin's comments yet as I've been quite busy working through my final semester since September. I can do a bit more work on the plugin over the break, but I think generally speaking it's in pretty good shape.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment