Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Build configuration missing. #1

Open
sheki opened this Issue Apr 12, 2011 · 10 comments

Comments

Projects
None yet
6 participants

sheki commented Apr 12, 2011

Is any configuration / build scripts /ant OR maven going to be released for building these tools any time soon?
Very much interested in looking at how Netty is used with thrift.

Thanks for open sourcing.

Contributor

jsirois commented Apr 20, 2011

We need to update the README and will be blogging about pants and sapling by ~end of May, but the build already works - but is custom (uses ant/ivy behind the scenes). You need python 2.6 and ant 1.8+ installed and then, from the root of the repo, you run:
$ ./pants tests/java/com/twitter/common:all-tests!

This will likely fail on the 1st run and explain steps for installing a python templating library (mako) dependency. After installing mako you should be good to go.

For example to compile all code and run all tests.
$ ./pants tests/java/com/twitter/common:all-tests!

You can learn a bit more using:
$ ./pants help
Tjis will list subcommands available.

$ ./pants help [subcommand]
This will give even more info about a particular subcommand.

In short - the repo is a monolithic source tree, but it produces many jars - vs a typical gradle/buildr/maven multi-module project where there would be a top-level project directory for each artifact-producing project. Currently pants provides 40 thin jars from its one source tree:
$ ./pants list --provides --provides-columns=artifact_id
com.twitter.common#application-action
com.twitter.common#application
com.twitter.common#application-http
com.twitter.common#application-modules
com.twitter.common#args
com.twitter.common#base
com.twitter.common#collections
com.twitter.common#inject
com.twitter.common#inject-timed
com.twitter.common#io
com.twitter.common#io-thrift
com.twitter.common#logging
com.twitter.common#dynamic-host-set
com.twitter.common#net-util
com.twitter.common#net-pool
com.twitter.common#net-http
com.twitter.common#net-http-handlers-text
com.twitter.common#net-http-handlers-string-template
com.twitter.common#net-http-handlers
com.twitter.common#net-http-handlers-thrift
com.twitter.common#net-http-handlers-time-series
com.twitter.common#quantity
com.twitter#serverset
com.twitter.common#stats-provider
com.twitter.common#stats
com.twitter.common#stats-jvm
com.twitter.common#stats-time-series
com.twitter.common#stats-util
com.twitter.common#thrift
com.twitter.common#jdk-logging
com.twitter.common#util
com.twitter.common#util-system-mocks
com.twitter.common#util-sampler
com.twitter.common#util-testing
com.twitter.common#util-caching
com.twitter.common#zookeeper
com.twitter.common#zookeeper-testing
com.twitter.common#stats-ostrich-adapter
com.twitter.common#service-thrift
org.apache.scribe#scribe_client

And some of these jars are very thin in and of themselves, leveraging internal and external jars:
$ ./pants files src/java/com/twitter/common/quantity
com/twitter/common/quantity/Amount.java
com/twitter/common/quantity/Data.java
com/twitter/common/quantity/Time.java
com/twitter/common/quantity/Unit.java

$ ./pants depmap -m src/java/com/twitter/common/quantity
com.twitter.common-quantity
com.twitter.common-collections
commons-lang-commons-lang-2.4
com.google.guava-guava-r08
com.google.code.findbugs-jsr305-1.3.9

The goal is to make it very easy to craft lean targets that export minimal APIs and compose these.

Any reason Maven isn't used here? It's pretty much the standard build system for open source java stuff.

Contributor

jsirois commented Jul 28, 2011

Yes - scalability on 2 fronts:
1.) What you see is a small slice of a much larger repo (using saplling: http://pypi.python.org/pypi/sapling https://github.com/jsirois/sapling) with >20 leaf projects and some common_internal libraries they share that are specific to internal twitter infrastucture. In total, using a maven-like build system, we'd have > 120 top level projects in a classic maven multi-project build. We were motivated by a strong desire to maintain a single source tree allowing for easy refactors across the global code base and easy discoverability of common code amongst 100s of developers scaling towards 1000s.
2.) We needed to support several languages 1st class. In particular - in this open source split you can see and use the excellent python support

The artifacts this build system exports are fully maven and ivy compatible, but a big shortcoming is that all the build documentation is still on internal wikis. This will be remedied at some point.

fern commented Jul 28, 2011

Any way to vote to change it to use maven? :)

Have you actually tried to compile and build with pants? It's exponentially faster than a Maven build would be with 120 projects in a multi-project build.

Stop complaining because you don't want to take the time to understand their tool.

Contributor

jsirois commented Jul 28, 2011

You could vote to change to maven or anything else if it fits the requirements described. In particular, besides the shortcoming of scant (non-existant!) docs right now on the public wiki for operating the pants build tool, what additional pain points are you encountering?

My pain point is that if you want people to contribute, use a standard build technology. I understand I'm saying this as an outsider though so take it like you wish.

Maven3 is quite a bit faster than maven2 these days, even supports OSGi. As a person who spent migrating the majority of eclipse.org's infrastructure to maven3 it doesn't suck that bad anymore. It has a large ecosystem where you can easily integrate with tools like Hudson/Jenkins, code coverage etc... It would be nice if this would be investigated in the future again.

Contributor

jsirois commented Jul 28, 2011

It does not support a single source tree though in any reasonable way afaict and does not support python.
I understand the pain point on contribution though and its on us to update the wiki with good docs.
That said - if you are an eclipse or intellij user, the following works well for development:
$ ./pants (idea|eclipse) tests/java/com/twitter/common:all -n common -c
See pants help (eclipse|idea) for more info.

cscotta commented Jan 9, 2012

Hi John,

I noticed about six months ago you mentioned plans to add documentation for your in-house build tool to the wiki, but I'm still not seeing anything here.

Probably just going to fork and start mavenizing a few things here.

Thanks,

– Scott

Contributor

jsirois commented Jan 9, 2012

Sorry for the large delay.

There will be a push this week with a much improved pants (no longer uses ant in the backend, uses compiler daemons for java and scala...). I'll be updating the wiki at that time with our internal pants wiki.

In the meantime the following may be helpful to you:

  • show all build commands
    ./pants help
  • get help on a build command
    ./pants help [command]
  • list all available build targets (targets are per library or test suite):
    ./pants list
  • generate a project for an ide
    ./pants (idea|eclipse) [targets]... -n myproject -c
  • compile all java targets and run the full java test suite
    ./pants tests/java/com/twitter/common:all

@jcrobak jcrobak referenced this issue in jcrobak/twitter-commons Nov 5, 2012

@benjyw benjyw Merge pull request #1 from jcrobak/classifier-support
Add generic classifier support for pants.
c2425a0

@jsirois jsirois referenced this issue in jsirois/commons May 1, 2013

jsirois Attempt #1 to fixup travis-ci config. c3a01ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment