Skip to content
A monorepo that holds all of Foursquare's opensource projects
Scala Python JavaScript Java Thrift Shell Other
Branch: master
Clone or download
Eric-Arellano and OniOni Add scripts to modernize to Python 3
These scripts were developed to modernize https://github.com/pantsbuild/pants as it drops Python 2 and can start to use Python 3-only features.

   - remove_builtins.py unravels part of the future library by removing all from builtins imports, which no-op in Py3. Its main innovations are the CLI interface and removing BUILD entries if possible.
   - update_headers.py removes from __future__ imports and # coding=utf-8 lines, which both no-op on Python 3.
   - update_decode_encode.py uses the default utf-8 encoding to simplify string calls to encode() and decode().
   - modernize_classes.py simplifies calls to super() and removes the unnecessary object base class, as all classes are new style in Python 3.

(sapling split of f017c31bc73d2c8345333cc7062de2b3c196a9cc)
Latest commit 332e7a2 Jun 13, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
3rdparty Update 3rd party map. Aug 20, 2019
build-support/fsqio pants build committing publish data for push of io.fsq#buildgen-emit-… Jun 6, 2019
scripts/fsqio Add scripts to modernize to Python 3 Aug 21, 2019
src Move `--userconfig` to webpack resolve as it's an npm option. Aug 21, 2019
test fix buildgen scalac plugin not handling _root_ (#14562) Jun 6, 2019
.gitignore Add pytest_cache to fsq.io gitignore (#6428) May 15, 2018
.travis.yml Generalize sapling projects under "opensource" and check-in fsqio bui… May 9, 2019
BUILD.opensource Pin scala minor version (#9262) Dec 13, 2018
BUILD.tools Drop a BUILD.tools in Fsq.io. Feb 16, 2017
CLA.md Move deployed files to consolidated directory. Jan 5, 2016
CONTRIBUTING.md Add a sapling quiver build (#14305) May 9, 2019
LICENSE.md Update copyright info in Fsq.io LICENSE. Mar 1, 2016
README.md Remove pom-resolve code and references (#3927) Feb 28, 2018
pants Add Fsqio build tooling to Fsq.io (#8728) Dec 13, 2018
pants-travis-ci.ini
pants.ini Add publishing config worked out in shading repo and docs (x2) (#15049) Jun 6, 2019
upkeep Support for WebpackSubsystem and add ivy.xml Apr 16, 2019

README.md

Foursquare Fsq.io

All of Foursquare's open source code in a single repo.

Build Status

All Foursquare code lives in a single repository, an architecture generally called a monorepo. Fsq.io is a subset of that internal monorepo. Fsq.io holds many of Foursquare's open source projects that had previously lived in their own separate Github repos. Foursquare contributes to a build tool specifically designed to work with monorepos called Pants. The entire Fsq.io repo is is built and tested by Pants.

Deploying directly from our monorepo has some nice advantages, for consumers of our open source projects as well as Foursquare itself. The entire repo is built daily by our CIs and internal contributions are open sourced automatically without the overhead of publishing. This repo will always contain the latest code that we use internally, all of the tools can be built just as we use them, directly from HEAD.

Projects include:

  • Exceptionator: An exception aggregator using mongodb.
  • Fhttp: A request building interface similar to scalaj-http for Finagle http clients.
  • Rogue: A Scala DSL for MongoDB
  • Spindle: A Scala code generator for Thrift
  • Twofishes: A coarse, splitting geocoder and reverse geocoder in Scala
  • and others.

Requirements

  • JDK 1.8 (1.8.0_40 preferred)
  • python2.6+ (2.7 preferred)
  • postgresql
  • monogdb server (required to pass some tests)
  • An increased number of file descriptors (we use 32768)

Internally we use OSX Yosemite or later. Other OS may work but are officially unsupported. (Unofficially, if building on Linux you should install python-dev, build-essential, and libpq-dev in addition to the above).

Pants build system

Pants is a build system that works particularly well for a source code workspace containing many distinct but interdependent pieces.

Pants is similar to make, maven, ant, gradle, sbt, etc. but pants pursues different design goals. Pants optimizes for:

  • building multiple, dependent things from source
  • building code in a variety of languages
  • speed of build execution

Pants is a true community project, with major contributions from Twitter, Foursquare, Square, among other companies and independent contributors. It has friendly documentation here, in this README we will just touch on how to compile and test the code.

Compiling and Testing

First Run

A good first run is to compile the repo and run every test.

 ./pants compile test src:: test::

Targets and BUILD files

Targets are adressable project or dependency level modules that can be built by Pants. BUILD files are configuration files that define targets that can be built by Pants. Each target has a name and can be built by running a Pants task against the target's name and location.

For example, Fsq.io's JVM projects live under src/jvm here. You can compile Rogue by running:

 ./pants compile src/jvm/io/fsq/rogue:rogue

Build and Test every project

Adding a :: to a path will glob every target under that location. So to compile every target in Fsq.io:

 ./pants compile src::

Similarly, to run all the tests, (after starting the mongodb server):

./pants test test::

Projects aspirationally have READMEs at the project root.

Acknowledgements

  • Thanks to the great community supporting Pants.
  • Fsq.io is split from commits to our internal monorepo by Sapling, a git porcelain tool by @jsirois.

Discussion

Please open an issue if you have any questions or concerns.

License

Apache License, Version 2.0 (Apache-2.0)

You can’t perform that action at this time.