New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CircleCI as an alternative to Travis CI #5767

Merged
merged 1 commit into from Jun 27, 2018

Conversation

Projects
None yet
6 participants
@bquorning
Member

bquorning commented Apr 11, 2018

I noticed in #5733 @bbatsov mention that

Perhaps it's time to start evaluating some alternative to Travis. Often I hear that Circle CI is more robust and less painful to use.

I agree!

This PR provides Circle CI configuration file. I have tested it from my fork of Rubocop, where a build finished in 11:49 which can be compared to 25:26 for the latest master build on Travis.

The plan is to keep running both CIs for a while, but eventually we will only run things in Travis that cannot easily be run on Circle – for example the builds against ruby-head and jruby-head.


Before submitting the PR make sure the following are checked:

  • Wrote good commit messages.
  • Commit message starts with [Fix #issue-number] (if the related issue exists).
  • Feature branch is up-to-date with master (if not - rebase it).
  • Squashed related commits together.
  • Added tests.
  • Added an entry to the Changelog if the new code introduces user-observable changes. See changelog entry format.
  • The PR relates to only one subject with a clear title
    and description in grammatically correct, complete sentences.
  • Run rake default or rake parallel. It executes all tests and RuboCop for itself, and generates the documentation.
@bquorning

This comment has been minimized.

Member

bquorning commented Apr 11, 2018

I tried adding bundler caching, but didn’t see any speed gains. Here’s the patch I tested:

diff --git a/.circleci/config.yml b/.circleci/config.yml
index e92f25cd6..d635d0083 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -3,8 +3,23 @@ version: 2
 steps: &steps
   steps:
     - checkout
-    - run: bundle install
-    - run: ruby .travis.rb
+    - run: bundle lock
+    - restore_cache:
+        keys:
+          - bundle-v1-{{ checksum "Gemfile.lock" }}
+          - bundle-v1-
+    - run: bundle install --path vendor/bundle
+    - save_cache:
+        key: bundle-v1-{{ checksum "Gemfile.lock" }}
+        paths:
+          - vendor/bundle
+    - run:
+        # When gems are not installed in the system, we need to change GEM_HOME.
+        # Shouldn't Bundler have a command for exposing this path?
+        name: Run tests
+        command: |
+          export GEM_HOME=$(bundle exec ruby -e 'print "vendor/bundle/#{Bundler.rubygems.ruby_engine}/#{Bundler.rubygems.config_map[:ruby_version]}"')
+          ruby .travis.rb
 
 spec_env: &spec_env
   environment:
@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Apr 11, 2018

I'm sold, but I'll leave the PR open for a while to hear other feedback (if any).

@Drenmi

This comment has been minimized.

Collaborator

Drenmi commented Apr 11, 2018

I use CircleCI for all work related CI matters, and like the experience. I use Travis for open source. I don't have a strong preference between the two.

I know people at Travis though, so will bring the feedback to them as well. 🙂

@SuperSandro2000

This comment has been minimized.

SuperSandro2000 commented Apr 13, 2018

The main point in this #5733 PR is invalid as Travis supports 2.2 instead of 2.2.0. And I'm pretty sure you can do everything with Build Stages.
Ref: https://docs.travis-ci.com/user/languages/ruby
https://github.com/SuperSandro2000/canuby/blob/master/.travis.yml#L5

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Apr 14, 2018

@SuperSandro2000 You can, but those are unlikely to be the latest versions, which is what we need. It used to work properly in the past, then at some point it stopped.

@SuperSandro2000

This comment has been minimized.

SuperSandro2000 commented Apr 14, 2018

works properly for me and the 2.5.1 update came really really fast.

@koic

This comment has been minimized.

Member

koic commented Apr 16, 2018

I have used CircleCI not much, but the reputation is not bad impression. On the other hand I have some know-how (Including bad know-how 😅) to Travis CI, but not in CircleCI.

When we have problems with Travis CI, it is difficult to distinguish between Travis CI image, Bundler or RubyGems. However, this is also a problem for the entire OSS community using Travis CI, and we are confronting each time ⚔️

My action may be delayed when problems arise in CircleCI, but this will also be a challenge 🔔

I will leave it to the judgment of the RuboCop community.

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 5, 2018

@bquorning Now that you're an org admin feel free to execute the remaining steps for the setup yourself. I think we should run for a while both TravisCI and CircleCI to make sure everything's good and afterwards we can just drop Travis completely.

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 25, 2018

Btw, I don't know if CircleCI supports this, but something we can do even now to improve the build process is a lot using Travis's build stages - I've been using them for a while on other projects (e.g. https://travis-ci.org/clojure-emacs/cider)

This basically allows you to split the build process in steps and the idea is to run first the quickest steps and halt the rest of the build when they fail. In our cases we can have some step to check the changelog format, the manual and the results of internal investigation. Then finally we can run the actual specs if all the fast (and common sources of mistakes) checks have passed.

Btw, I also wonder do we still need to run those ascii_specs. They basically make the build time double and I don't recall them unveiling any problems recently.

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 25, 2018

CircleCI has workflows which can do the same thing. I (mis)use them in rubocop-rspec to start the slowest job (jruby) in the first job batch, so it’ll finish around the same time as the 4th batch of faster jobs.

Splitting jobs and perhaps dropping ascii_specs is out of scope of this PR, I think.

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 25, 2018

Yeah, you're right. I was just adding here some random (somewhat related) thoughts.

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 25, 2018

It seems the Docker images circleci/ruby:latest and circleci/jruby:latest are built from the latest stable release, and not from ruby-head.

What should we do about this issue? Should 1) I drop these tests, 2) install ruby-head on the fly, or 3) make (and maintain) a Docker image?

@bquorning bquorning force-pushed the bquorning:circle branch from 4684ab0 to aeb80e6 Jun 25, 2018

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 25, 2018

Let's just drop those tests. They add very little value - frankly, I never pay them any attention.

Maybe someone can suggest to Circle to start building head images, but I certainly don't want us to waste time on this ourselves.

@bquorning bquorning force-pushed the bquorning:circle branch from aeb80e6 to df446a1 Jun 25, 2018

@koic

This comment has been minimized.

Member

koic commented Jun 25, 2018

I had overlooked an important mistake 💦 I'm not familiar with CircleCI, but the ruby-head and jruby-head tests running on Travis CI were missing in this CircleCI setting.

I'm looking at both ruby-head and jruby-head in Travis CI. Especially the release of ruby-head is once a year Christmas. When problems overlap during the year, the complex problem is hard to solve. Problems that failed with ruby-head can also be problematic in Ruby itself and other OSS community. I feel that knowing these failures early will lead to the development of the entire Ruby community.

So. I emphasize the ruby-head and jruby-head tests, and if they can not run it thinks that maintaining Travis CI is better.

Actually now I can see that there is a problem with Code Climate in ruby-head. And I can learn that there are no other problems via Travis CI There.

@bquorning bquorning force-pushed the bquorning:circle branch from df446a1 to 346100c Jun 25, 2018

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 25, 2018

@koic In an earlier version of this pull request, I was testing with the Docker images circleci/ruby:latest and circleci/jruby:latest, but I noticed that they are built from the latest stable release, and not from ruby-head.

I have asked CircleCI if they are interested in maintaining ruby-head docker images that we can use: https://discuss.circleci.com/t/ruby-head-and-jruby-head-docker-images/23180

If they will not do that, I noticed that they maintain a circleci/ruby:rc image, which I believe will always be the latest release candidate. Would that be as useful as ruby-head image?

@koic

This comment has been minimized.

Member

koic commented Jun 26, 2018

@bquorning Thank you for asking them about the Docker image.

Would that be as useful as ruby-head image?

Unfortunately, I think that testing on release candidates (RC) is not useful compared to trunk (ruby-head).

The release candidate is a snapshot of the trunk of a specific point. And release candidates are published only few times a year.
On the other hand, since the trunk is updated daily, it is always ahead of release candidates. So my concern is testing with trunk.

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 26, 2018

Also here: docker-library/ruby#222

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 26, 2018

Btw, as workaround we can just move everything but the snapshot builds to Circle CI. Anyways, hopefully the situation with the missing images will be resolved soon and we won't have to worry about that.

@bquorning bquorning force-pushed the bquorning:circle branch from 5217aec to 47adb7c Jun 26, 2018

@bquorning bquorning changed the title from Use Circle CI instead of Travis to Use CircleCI as an alternative to Travis CI Jun 26, 2018

Add CircleCI configuration
We want to try out CircleCI and see if it is not only faster, but also more
robust and less painful to use than Travis CI. Being able to use Docker images
should at least provide some stability that we have not previously experienced.

Compared to the Travis CI configuration, we are missing the ruby-head and
jruby-head tests in CircleCI.

@bquorning bquorning force-pushed the bquorning:circle branch from 47adb7c to 1018464 Jun 26, 2018

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 26, 2018

From time to time, a jruby job doesn’t completely finish, but is killed and fails. E.g. https://circleci.com/gh/bquorning/rubocop/387

I’m hesitant to merge this PR until I have found out why that happens.

# CircleCI container has two cores, but Ruby can see 32 cores. So we
# configure test-queue.
# See https://github.com/tmm1/test-queue#environment-variables
- TEST_QUEUE_WORKERS: 2

This comment has been minimized.

@bquorning

bquorning Jun 27, 2018

Member

These ENV variables should probably be moved into the CircleCI env, since TEST_QUEUE_WORKERS applies to all builds, and JRUBY_OPTS will be ignored by the MRI builds and only used by JRuby builds.

@bquorning

This comment has been minimized.

Member

bquorning commented Jun 27, 2018

Adding 2 ENV variables (present in .travis.yml) seems to do the trick for jruby builds:

# By default we have 2 CPUs and 4GB ram available
# https://circleci.com/docs/2.0/configuration-reference/#resource_class
JRUBY_OPTS="--debug -J-Xmn1024m -J-Xms2048m -J-Xmx2048m"
# CircleCI container has two cores, but Ruby can see 32 cores. So we
# configure test-queue.
# See https://github.com/tmm1/test-queue#environment-variables
TEST_QUEUE_WORKERS=2

I suggest that we add them to CircleCI’s environment instead of polluting the configuration file with them – so I have removed them from this PR’s diff.

And with that, I believe this PR is ready to merge.

If anyone has further tips for configuring the JRuby memory limits, please let me know.

@bquorning bquorning force-pushed the bquorning:circle branch from a79ca75 to 1018464 Jun 27, 2018

@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 27, 2018

I suggest that we add them to CircleCI’s environment instead of polluting the configuration file with them – so I have removed them from this PR’s diff.

Fine by me.

@bbatsov bbatsov merged commit 830382f into rubocop-hq:master Jun 27, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@bbatsov

This comment has been minimized.

Collaborator

bbatsov commented Jun 27, 2018

And a big thanks for working on this! 🙇

@bquorning bquorning deleted the bquorning:circle branch Jun 27, 2018

@deivid-rodriguez deivid-rodriguez referenced this pull request Sep 11, 2018

Merged

Test Ruby release candidates on CircleCI #6275

5 of 8 tasks complete

@bquorning bquorning referenced this pull request Sep 13, 2018

Closed

ruby:head image #222

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment