Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Announcement: Calabash 2.0 #55

Open
jmoody opened this issue Apr 29, 2015 · 65 comments

Comments

@jmoody
Copy link
Member

commented Apr 29, 2015

Calabash 2.0 is a merge of the iOS and Android APIs.

Why?

  • A unified API will make cross-platform testing much easier.
  • APIs are difficult to design. Now that we've had several
    years of experience with hundreds of different applications
    on both Android and iOS, we believe we've created an API that is
    easier to use and understand.
  • Both projects have accumulated a lot of behaviors that are
    either broken, no longer relevant, wrong-headed, or dangerous.
    Calabash 2.0 is our opportunity to fix what is broken and
    deprecate what is unnecessary.

Goals

  1. Wherever possible, the Calabash 2.0 API will be cross-platform.
  2. The API and tools will have a high level of test coverage.
  3. The Calabash toolchain will be semantically versioned.

Features

  • Better wait behaviors.
  • Better gestures interface.
  • Better query results.
  • Fewer environment variables.
  • More and better documentation.
  • More example projects.
  • Dropping support for Ruby 1.9.
  • Cucumber 2.0 support.

FAQ

Q1 Are the Calabash 2.0 and Calabash 0.x.x APIs compatible?

The short answer is, no. We've redesigned the API from the bottom. All of the API will be familiar, but many method signatures have changed. You will almost certainly have to refactor some of you code. Ouch! Yeah, we understand, and if it makes you feel better, we're not happy about it.

Q2 Why are the Calabash 2.0 and Calabash 0.x.x APIs not compatible?

There were many inconsistencies between the Android and iOS APIs. We analyzed our options and found that if we made a backward compatible API, we'd be repeating many mistakes and not making much improvement.

Q3 When will Calabash 2.0 be released?

Developers get burned by answering this question all the time. Estimating is hard and no matter how much experience you have, your chances of being correct are very low. Our plan is to start making pre-releases very soon.

Q4 Can I see the new API?
Q5 Can I comment on the API?

The best place to comment is on GitHub, either with a pull request or an issue.

Q6 How will support be handled?

With Calabash 2.0, we plan to merge our support channels. We don't yet know what that will look like. The Google Groups are a difficult place to do support; questions and answers cannot be curated, older posts with incorrect information cannot be deleted, and the support formatted code is almost nil. We have been discussing tools like Vanilla Forums and discuss.io. We'd like to hear from you. Do you know of an awesome support tool? We'll open topics on the Calabash iOS and Android groups where you can weigh in with opinions.

Q7 Why 2.0?

We feel like Calabash is ready for a 2.0 release. This API will feel like a dramatic departure from the old because you will need to refactor your test suites. And we can't release a 1.x version because RubyGems .org already has a Calabash version 1.2.6.

@JaniJegoroff

This comment has been minimized.

Copy link

commented Apr 30, 2015

@jmoody - That's awesome news! 👍

I'm delighted to see that coding style guide is taken in use. Since style guide is based on Rubocop style guide (https://github.com/bbatsov/rubocop) I would highly recommend to utilise it's capabilities (static code analysis, custom configurations). There are currently over 1200 offences in the development branch (including test code) and starting to fix them sooner rather than later would be essential. I'm happy to contribute if you agree.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Apr 30, 2015

@JaniJegoroff

I am not opposed to using a code quality tool. However, I only want to use it if is tied into Travis CI. Can you wire something like this up?

Also, 1200 offences seems how. How configurable is the analyser? Our style guide is not exactly rubocup.

@NM-Clio

This comment has been minimized.

Copy link

commented Apr 30, 2015

+1 for discuss.io. I haven't used Vanilla Forums.

+million for shifting away from Google Forums.

@bootstraponline

This comment has been minimized.

Copy link

commented May 4, 2015

I recommend discourse as a Google groups alternative. It's worked out well for appium.

@LiohAu

This comment has been minimized.

Copy link

commented May 7, 2015

Good news :)

About the support, I also recommend to move from google groups to discourse.

When you say "Wherever possible, the Calabash 2.0 API will be cross-platform.", I assume that "Wherever possible" is used for cases like physical back button (and long press maybe?) which are specific android features.

What I would really appreciate:

1/
Given that a lot of testers are not primarily developers
And that it has already been hard to train them for web testing
And that, usually, selenium is the web driver used (because it is the most used)
When testers write their new tests cases targeting mobile apps
Then they should not have to learn a completely new API or language

So basically, if the calabash API could extend/mimic the selenium API (if another is more widely used, then extend that one).

2/
Calabash iOS / Android is mixing two components: automation driver, and cucumber integration, so this tool chain is not possible :

(Specflow => .NET | Cucumber => Ruby) <-> Calabash | Appium | Selenium API | Any other web/mobile driver <-> (Connected device | Any test cloud) <-> App (optionally embedding calabash server)

Having a specific part of calabash being the automation driver, and another part related to cucumber (that could be replaced by a specflow part), would be awesome.

@bootstraponline

This comment has been minimized.

Copy link

commented May 7, 2015

I agree about using the selenium API (for success stories with this approach, see appium, selendroid, ios driver). It's language agnostic and allows reusing knowledge from existing web tests.

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

Better wait behaviors.

It'd be great if calabash moved away from the Timeout API. It's super flaky. The selenium gem's approach to timeouts is much better.

@TobiasRoikjer

This comment has been minimized.

Copy link
Contributor

commented May 8, 2015

@bootstraponline Why do you find the timeout api to be flaky?

An advantage of the Timeout API is that it will still timeout, even if the given block never returns. The "selenium method" would keep running.

Imagine yield never returning.

        until Time.now > end_time
          begin
            result = yield
            return result if result
          rescue *@ignored => last_error
            # swallowed
          end

          sleep @interval
        end
@krukow

This comment has been minimized.

Copy link
Member

commented May 8, 2015

Actually @bootstraponline is correct there are well known problems with the timeout API and I agree we should revisit the approach. However, @TobiasRoikjer is also correct that the selenium approach is naive. I'm wondering if we need two approaches: conceptually cooperative_timeout if we trust the the block will not run for a long time and timeout if we don't know the block.

Or perhaps there is a better unified approach (surely someone in ruby land must have looked at this before)

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

Selenium is widely used and the wait code has been working properly in production for 5 years. When I used the timeout api in appium, it was flaky. After researching and finding that language implementors such as jruby recommend against it, the decision was straightforward.

The timeout issue is one of the reasons I moved away from calabash. Hopefully this area is improved in v2.

@krukow

This comment has been minimized.

Copy link
Member

commented May 8, 2015

@bootstraponline that's exactly what I mean by "there are well known problems with the timeout API". I also understand that the approach you're referencing works well as long as the block that is yielded is well-behaved. That's why I think we need either two approaches or a unified better approach

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

@krukow I agree with your post. I was responding to TobiasRoikjer.

@TobiasRoikjer

This comment has been minimized.

Copy link
Contributor

commented May 8, 2015

@bootstraponline @krukow Is the Timeout API really flaky if used correctly? Reading about Timeouts in jRuby, they appear to have been fixed as well.

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

(surely someone in ruby land must have looked at this before)

When I researched this in the past, there was no shortage of abandonware attempts at resolving the issue.

@bootstraponline @krukow Is the Timeout API really flaky if used correctly?

The timeout API in MRI is fundamentally broken. That's been my experience twice over, once with calabash and then again with appium. Timeout being broken is widely documented online.

@LiohAu

This comment has been minimized.

Copy link

commented May 8, 2015

I have two wishes that I did not mention in my previous post.

When testing web sites the drivers knows all the elements on the page, elements having visibility: hidden, or display: none, but also the elements that are outside of the browser view. This is because the browser has a full DOM.
On iOS (and I guess that the issue also exists on android, but I don't know), elements that are not displayed on the screen, does not exists yet. For instance, if you try to touch the 7th row of a tableview containing 10 rows having an height of 100px on a 480px screen, the API will fail, because the devices only has 5 rows in memory at the time of the query.
This problem is created by the recycling system of the UICollectionView/UITableView (that's why I said that I don't know if the problem exists on android).
As a test developer, I can scroll the views by myself to make these components appear, but it's really painful to develop.
So I think that the API should handle this for us.

1st solution:
When calling query/touch methods, the API should accept a parameter that would be a list of UI components that can be scrolled (and in which direction they can be scrolled) in order to discover components that are potentially non existing at the time of the query. Howerver doing UI things like scrolling views to be able to query the targeted view could trigger specific application behaviors that are not expected for the test. In my opinion, it is the only inconvenient of that method.

2nd solution:
Since calabash uses an obj-c server in the app, we can call UITVC/UICVC delegate methods to obtain more informations about the number of cells, or ask these delegates to creates the missing cells to be able to execute query on them.
This solution should create less side effects than the 1st one. But this is also less generic, because that would only works with view controllers that are handled by the calabash server (It won't work if I create my own clone of UITVC for example). And last inconvenient, it won't work on android since there's is no server in the app.

Talking about the android server leads me to my 2nd wish, why isn't there a server inside the android app? Sometimes an automated test cannot be developed without the backdoor method.

@TobiasRoikjer

This comment has been minimized.

Copy link
Contributor

commented May 8, 2015

@LiohAu To answer your last question: There is an Android server. Because of the way Android instrumentation is done, we can have the server as a separate apk, which does not require you to modify your application. The server in Android is fully capable of interacting with your application as if it was inlined in your code. The backdoor method is available for Android as well.

@LiohAu

This comment has been minimized.

Copy link

commented May 8, 2015

@TobiasRoikjer Ok I did not know, so I only have 1 wish remaining (in this post, but there is also my first post :D)

@jmoody

This comment has been minimized.

Copy link
Member Author

commented May 8, 2015

@LiohAu re: 2nd solution - This is what the server does already.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented May 8, 2015

@bootstraponline

The timeout API in MRI is fundamentally broken

How does this manifest? In particular, how does it manifest in Calabash?

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

@jmoody I recommend discussing with @krukow if you don't believe the evidence that the timeout API has issues on MRI. I don't want to spend my time arguing on github.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented May 8, 2015

@bootstraponline

I didn't realize we were arguing. I am information gathering. I am not disputing anything. I want to understand how this manifests in Calabash so it can be addressed.

@bootstraponline

This comment has been minimized.

Copy link

commented May 8, 2015

I've linked to a number of sources of information. The Appium Ruby code originally used the same approach as calabash. Users complained about wait flakiness and I experienced the same thing. I tracked the problem down to use of the Timeout API and discovered this was a well known issue. After switching to the Selenium wait approach (which I agree only makes sense for well behaved blocks) the stability improved and the complaints stopped.

My goal in commenting here was to point out a potential area for improvement. I think I've accomplished that. I don't have any additional information to add. I like Xamarin and wish the best of luck to the calabash v2 project.

@LiohAu

This comment has been minimized.

Copy link

commented May 9, 2015

@jmoody I think you are talking about is done in the LPCollectionViewScrollToItemWithMarkOperation.m / LPScrollToRowWithMarkOperation.m for scrolling to a specific mark and in the UIScriptASTWith.m that supports the indexPath keyword.
But this is a partial implementation, I should be able to make : query 'label marked:"X"' and if that label is in a UITableViewCell / UICollectionViewCell / WhateverCustomCell that is not yet allocated, then I should have a way to get a result from my query.

@mvemjsun

This comment has been minimized.

Copy link

commented May 10, 2015

Could I suggest that when developing official documentation around Calabash 2.0 it would be good to have a section or links to

  1. Calabash Good practices, the good practices could cover both the test code that is being developed also what conventions/patterns should be followed while developing the apps themselves so that they can make the App more testable . All testers may not have in-depth knowledge of how the Apps are build but if we know of the areas that are important for better integration with Calabash then we could nag the developers.
  2. Some design patterns (Apart from the obvious Page Object that calabash helps implement) that people have discovered and used often that could be used for calabash-cucumber-ruby projects.

Apart from that I would like if there can be an API that will help to access the device console logs, I currently use idevicelog, from the command line. It would be good to have an adapter of some sort to these command line utilities.

For logging via Calabash::Cucumber::Logging it would be good to have standard logging levels as described at Logging Levels.

not sure if there are some of these already available though ? I have been using Calabash for a few months now and find it useful in my automation tasks.

@Goginenni

This comment has been minimized.

Copy link

commented May 12, 2015

Good to see Selenium API support in 2.0. Most people argue about Calabash is Ruby and Cucumber dependency. Would be better if it is moving in the direction where it can be used without cucumber and any other language

@TobiasRoikjer

This comment has been minimized.

Copy link
Contributor

commented May 13, 2015

@bootstraponline Calabash iOS and Android had potential issues with timeouts (and potentially a lot of other things) because of improper rescuing of Ruby Exception (which should almost never be done). These issues have been fixed on master. This could lead to unexpected behaviour. As for Timeout.timeout itself, we do not believe it to be faulty and broken - as long as it is used correctly. E.g. do not timeout a block with an ensure clause.

@vhugogarcia

This comment has been minimized.

Copy link

commented May 15, 2015

Probably the most important desired feature at least for me, is to find the most easy way to grab the components IDs, so we can create the rule tests faster.

In the last version, just see how much you spent doing that... Hope, there is a way to do that just easier and faster.

@Goginenni

This comment has been minimized.

Copy link

commented Jun 25, 2015

Another idea we had at Badoo is replacing Robotium with Espresso for Calabash-android backend framework. We started changing the backend framework code and able to run 70% of out test cases using Espresso. As Espresso claims, we saw the slight increase ( Not super fast as we thought in the beginning ) in the speed of test cases execution but still few issues to resolve and few client functions has to change due the way Espresso works. Is anyone thinks this is good to do then we will share the code and discuss about this.

@bootstraponline

This comment has been minimized.

Copy link

commented Jun 25, 2015

I'd be very interested in seeing the code for the Espresso backend. In appium we've also looked at Espresso and decided google's droiddriver is easier to integrate with.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Aug 21, 2015

@ericzered

This comment has been minimized.

Copy link

commented Aug 26, 2015

Oh ok, thanks !
That's why I can't use "calabash-ios console" too...
I was only using the console like this, what is the other way to use calabash without command line ?
Thanks

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2015

@ericzered

The console is working for both iOS and Android - and it is much improved.

$ be calabash console /path/to/Your.app
$ be calabash console /path/to/Your.ipa
$ be calabash console /path/to/Your.apk

# If the application is installed, it will be re-installed.
> install_app

# Will start the application.
> start_app

See the docs for more information about the app life cycle methods.

@ericzered

This comment has been minimized.

Copy link

commented Sep 4, 2015

Hello,
I've installed "calabash 2.0.0.pre3" but to run my cucumber test, I need the gem "calabash-cucumber".
So, because I can't find the pre-release for this gem, I still have :
"Unable to activate calabash-cucumber-0.14.3, because cucumber-2.0.0 conflicts with cucumber (~> 1.3.17) (Gem::ConflictError)"
How can I run a Calabash cucumber tests and keep cucumber 2.0.0 ? (I need this version on my machine).
Thanks a lot !

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Sep 4, 2015

Why do you need the calabash-cucumber gem?

Are you using a Gemfile? What are the contents?

Calabash 2.0 does not depend on Cucumber at all.

@ericzered

This comment has been minimized.

Copy link

commented Sep 4, 2015

I have a cucumber project with Calabash to run on a specific machine.
My project start like this :
@calabash_launcher = Calabash::Cucumber::Launcher.new
unless @calabash_launcher.calabash_no_launch?
@calabash_launcher.relaunch
@calabash_launcher.calabash_notify(self)

So because I saw that calabash 2.0 will support "Cucumber 2.0" as it said in the features above, I'm trying to find a pre-version to run my testrun project.
Thanks

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Sep 4, 2015

@ericzered We are getting off topic. Please open a new issue.

@sleekweasel

This comment has been minimized.

Copy link

commented Sep 8, 2015

Is there any chance that Cucumber 2.0 could be tweaked to cope with parallel test execution?

Android devices are happy to run separate tests at the same time; web browsers can run in multiple processes and profiles to the same end. iOS is unofficially (6.4) able to host multiple test devices if run under multiple users via ssh; it's only Windows phone that (apparently) can't do multiple devices at all.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Sep 8, 2015

@sleekweasel I am assuming you mean Calabash 2.0. :)

The Calabash 2.0 architecture was designed with multi-device testing in mind.

iOS is unofficially (6.4) able to host multiple test devices if run under multiple users via ssh

We haven't explored this at all.

Android

Should be possible out of the box with Calabash 2.0. We have no examples however.

@tejasv02

This comment has been minimized.

Copy link

commented Sep 17, 2015

Thats a Great move.
With calabash cross platform my code sharing has increased incredibly.

I have managed to fork the repo and take a look it look amazing ..
Thanks for all the effort

@prat23k

This comment has been minimized.

Copy link

commented Oct 20, 2015

When we can expect official Calabash 2.0 release, we tried xamarin test cloud,its awesome devices are very stable.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Oct 20, 2015

@prathimak

I don't have a release date.

@prat23k

This comment has been minimized.

Copy link

commented Oct 20, 2015

@jmoody thanks

@dnadan

This comment has been minimized.

Copy link

commented Dec 2, 2015

Hi @jmoody ,

Do you have anything new to share for Calabash 2.0 about your investigation in calabash/calabash-ios#876? Or is that further down the line?

Many thanks.

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Dec 2, 2015

No updates on Calabash 2.0 and nothing new on integration XCUITest except that it is definitely the future as Apple has deprecated UIAutomation.

@dnadan

This comment has been minimized.

Copy link

commented Dec 2, 2015

@jmoody Ok, thanks for the info.

@LiohAu

This comment has been minimized.

Copy link

commented Dec 2, 2015

Is there a future for the calabash client since the Xamarin.UITest client is now free for all ?

@bootstraponline

This comment has been minimized.

Copy link

commented Dec 2, 2015

Where did they announce that Xamarin.UITest is free?

@jraczak

This comment has been minimized.

Copy link

commented Dec 3, 2015

@bootstraponline We announced this as part of Xamarin 4 two weeks ago. You can read about it here: https://blog.xamarin.com/introducing-xamarin-4/

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Dec 3, 2015

@LiohAu Yes, there is a future for Calabash. :)

@LiohAu

This comment has been minimized.

Copy link

commented Dec 3, 2015

Well that would be awesome if they could integrate a gherkin "step" explorer in the XTC Recorder. So you can bind 1 or multiple recorded actions to a gherkin step.

@paulpatarinski

This comment has been minimized.

Copy link

commented Feb 8, 2016

Any plans on making iOS testing similar to calabash-android (not requiring an embedded http-server), therefore making it possible to test Release builds?

@llopezTopps

This comment has been minimized.

Copy link

commented Feb 17, 2016

Any plans on making/supporting a calabash for Unity applications?

@smartfaceTester

This comment has been minimized.

Copy link

commented Jul 11, 2016

Hello,
I am pretty new on Calabash. Is it possible to use Calabash 2.0 with real iOS devices? I am asking because I got this message.

RuntimeError: Could not find a simulator with a UDID or name matching 'c11b68cf24981e201532e220e2d30bc633595c55'

Thanks

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Jul 11, 2016

Yes, it is possible.

Unfortunately, Calabash 2.0 is out of step with run_loop.

My advice is to start with Calabash 0.x.

On Monday, 11 July 2016, smartfaceTester notifications@github.com wrote:

Hello,
I am pretty new on Calabash. Is it possible to use Calabash 2.0 with real
iOS devices? I am asking because I got this message.

RuntimeError: Could not find a simulator with a UDID or name matching
'c11b68cf24981e201532e220e2d30bc633595c55'

Thanks


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#55 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAccuEx1i0QLai3Fjyyjpf1V7D9qm9kDks5qUdrAgaJpZM4ELrtG
.

@fabb

This comment has been minimized.

Copy link

commented Oct 8, 2016

Is there some migration guide from 0.x to 2.0?

@jmoody

This comment has been minimized.

Copy link
Member Author

commented Oct 10, 2016

@fabb Not yet.

@fabb

This comment has been minimized.

Copy link

commented Oct 12, 2016

Seems like the Appium guys are working actively on supporting XCUITest, maybe Calabash can profit from their sharing of implementation details:
appium/appium#5225

@JaufretDemaries

This comment has been minimized.

Copy link

commented Nov 14, 2016

@jmoody I've start with calabash-cucumber and calabash-android separately and got a bunch of questions.

Is the 2.0 project more current than the two separate calabash-cucumber and calabash-android for cross platform ?

Is there troubles with last version of iOS or Android to use it with both emulators and physical devices ? Or any pro / cons ?

Thank you for your answers.

@jamespullar

This comment has been minimized.

Copy link

commented Feb 17, 2017

Will the unification still support ABase and IBase classes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.