Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Introducing Cukes #431

Merged
merged 156 commits into from
@steveklabnik

Please note this is not merge-able yet, this is mostly for discussion purposes.

Hey all! One of the first things I've been tasked with is improving the API specification. Over the weekend I asked around my network, did some research, and spiked out an idea, and I wanted to run it all past you.

Requirements

As always, the first step is to gather requirements. Here's what I gather we need for the API spec:

  1. A way to define what each aspect of the API does.
  2. An automated way to turn those definitions into executable tests.
  3. A way to run those tests regularly to ensure conformance.

All three of these are pretty straightforward, but might as well spell them out.

  1. These specifications must be easy to understand by a large variety of people.

This one is complicated. There are two kinds of concerns here: {in,out}side of Balanced, and across software language groups.

The first is the in/out part. Basically, while we have a very clear idea of what everything does, people who are new to Balanced won't. Just today I got an email from someone, asking if Balanced was appropriate for them, when their marketplace was basically identical to Rent My Bike. I'm happy to answer those kinds of questions, but we need something that will be easy for people who don't live and breathe payments to grok.

Secondly, software language groups. If we write our specs in Python, Pythonistas will be able to understand it, but Haskellers might not. Well, they probably will, but you know what I mean. Everyone wants their preferred language, but our product is an API, and it's language agnostic. Therefore, we need something that's not incredibly closely tied to a particular software language.

  1. The specification has to facilitate collaboration on what the specification actually does.

This is related to #4, but is a bit different. It's not enough that the spec is legible to everyone, but that it's easy to bikeshed. ;)

What we have

Right now, the API spec looks largely like this: https://github.com/balanced/balanced-api/blob/revision1/scenarios/cards/tokenize_without_secret.yml

It's a bunch of YAML files, with a totally custom, bash/Python runner.

This situation is good in that it's language agnostic, but bad in that it's 100% custom, so it's not natively understood by anyone, and while it communicates to some degree of what the specs do, it's almost 100% code. It took me some time to be able to actually understand what's going on, which will always happen, of course, but it's still hard to turn into actual words, which impedes #4 and #5.

Solution

I've spiked out a temporary solution based on a tool called "Cucumber." Before I tell you more, let's compare:

Current: https://github.com/balanced/balanced-api/blob/revision1/scenarios/cards/tokenize_without_secret.yml

Cucumber: https://github.com/balanced/balanced-api/blob/cukes/features/cards/tokenize.feature

The 'current' has two features, and the cucumber only has one, so ignore past line 18 on 'current.'

Cucumber, if you haven't heard of it, is a tool for collaboration between stakeholders. The idea is that you write high level descriptions of what you want your software to do in psudo-English ("Gherkin"), and then that gets transformed into code to actually run and test those descriptions.

While I've implemented the scenarios in Ruby, you can write the 'code' aspect of the tests in a variety of languages. You can also write the features in a variety of languages, too. For example: https://github.com/cucumber/cucumber/blob/master/examples/i18n/el/features/addition.feature

As for how this gets turned into code, I have two versions:

JSON Schema: https://github.com/balanced/balanced-api/blob/cukes/features/step_definitions/card_steps.rb

Assertions: https://github.com/balanced/balanced-api/blob/b91d4264eaace33373606191ef4d32ca1372ac47/features/step_definitions/card_steps.rb

Since we already assert against JSON Schema, I think sticking with that makes sense, as it'll be easier to port.

I actually think the "200 code" assertion may be too low level.

Pros

Basically, this is the exact use case for Cucumber. Because the specifications are written in English, we help make it easier for people to understand, as well as making it easier from people to collaborate across language boundaries. I have some issue with English being the de facto human language that technologists communicate in, but it is what it is, at least for now. This solves #4 and #5, our biggest problems, nicely.

Furthermore, collaboration can be done on the high-level language, without worrying about the details. For example, Matin had some concern over my wording of the last part of that scenario, because 'that's not exactly what that does.' Excellent, we have some sort of communication error here. Cucumber's high-level descriptions bring that kind of thing out, and people can submit pull requests with just the Gherkin, and we can add the implementation ourselves.

I wonder if "elevator pitches for your code" is a good way of describing Cucumber. Maybe I've been in San Francisco too long. ;)

Cons

Cucumber is... not a tool without some controversy. It was originally described as an 'acceptance testing' tool, and so people got into 'test mode' and started writing tests. Cucumber is very bad when used in this way. Eventually, even some helpers were removed [1] to discourage writing tests at a low level. Basically, if you're not writing very high-level descriptions, you are Cuking It Wrong. ;)

Cucumber can be slow. There's the whole Gherkin-code 'compilation' step, so it can take longer than other frameworks. This is mitigatable, but we're already planning on making HTTP requests against our API for our tests, so they'll be slow no matter what the framework is.

As I mentioned above, some people really, really dislike cukes. I think that's mostly due to having terrible Cucumber suites, people have developed some scars. This can be overcome, but there may be some initial 'eww.' There may be with our specs as of right now. ;)

Other options

Basically, if this is a bad idea, we should just move to some sort of very basic unit testing library in a not-to-unusual language. I'd basically use Python and Requests to do this, since it's what we're most familiar with overall. I think that the downsides of being language specific is pretty big, though.

Conclusion

I'm sure I've forgotten a few things, but I'd love to hear feedback, good and bad. I think this is the right move, but I don't always have all the right answers... I would especially love feedback from @mattwynne and @aslakhellesoy .

1: http://aslakhellesoy.com/post/11055981222/the-training-wheels-came-off

Scenarios

CRUD tests

Here's all the cases we need to cover to merge. From https://docs.balancedpayments.com/1.1/api/ :

API KEYS

  • Create a Key
  • Get a Key
  • List API Keys
  • Delete API Key

BANK ACCOUNT VERIFICATIONS

  • Create a Bank Account Verification
  • Get a Bank Account Verification
  • Confirm a Bank Account Verification

BANK ACCOUNTS

  • Tokenize a Bank Account (Direct)
  • Get a Bank Account
  • List Bank Accounts
  • Delete a Bank Account
  • Credit a Bank Account
  • Debit a Bank Account

CALLBACKS

  • Create a Callback
  • Get a Callback
  • List all Callbacks
  • Delete a Callback

CARDS

  • Tokenize a Card (Direct)
  • Get a Card
  • List All Cards
  • Update a Card
  • Deleting a Card
  • Debit a Card

CREDITS

  • Create a Credit
  • Get a Credit
  • List All Credits

CUSTOMERS

  • Create a Customer
  • Get a Customer
  • List Customers
  • Delete a Customer
  • Associate a Card
  • Associate a Bank Account

DEBITS

  • Create a Debit
  • Get a Debit
  • List All Debits
  • Update a Debit
  • Refund a Debit

EVENTS

  • Get an Event
  • List all Events

REFUNDS

  • Create a Refund
  • Get a Refund
  • List All Refunds
  • Update a Refund

REVERSALS

  • Create a Reversal
  • Get a Reversal
  • List All Reversals
  • Update a Reversal

non-crud tests

That's exhaustive. Let's do this.

@mattwynne

I like it!

You'll have to see how it plays out to stick to this kind of high-level language in all of your scenarios. Some people might want to see more detail, yet not want to read the Ruby code underneath in the step definition. You can either address this with hand-crafted user docs (which could go in the Gherkin), or you could consider having one scenario per feature that uses more detail and has concrete examples of the JSON request / responses.

I'm interested to see how this develops.

@tef
tef commented

I think as it stands, your requirements are manufactured to want Cucumber, so let's look at them:

A way to define what each aspect of the API does.
An automated way to turn those definitions into executable tests.
A way to run those tests regularly to ensure conformance.

The second step is the one you'll have to justify, and I'm not convinced.

. Basically, while we have a very clear idea of what everything does, people who are new to Balanced won't. Just today I got an email from someone, asking if Balanced was appropriate for them, when their marketplace was basically identical to Rent My Bike

Writing these tests in pseudocode won't make people read them. (Unit) tests themselves do not give big picture views of your code, and should not serve as documentation, or example code.

If you have a comprehensive test suite, would you expect people to browse through them one-by-one to see if their use case works?

Secondly, software language groups.

There is an xkcd cartoon here, but instead of using a language some people know, you're going to use a language no-one knows. Although some tests become 'readable', the maintenance of tests becomes labourious. Frequently for each line of english you have to write a custom regular expression to parse it. Your test runner won't be language agnostic either.

Therefore, we need something that's not incredibly closely tied to a particular software language.

Except the implementation will be, and you will need to rewrite the ad-hoc regex rules in each language if you want to avoid this. This is a huge burden for tests and maintenance.

Basically, this is the exact use case for Cucumber

The use case of cucumber (afaik) was so that clients signing off on a project could agree to pages and pages of legalese that was executable, to enable the customer to both understand and specifiy what they want.

This is not your use case. Your use case as presented is "We need better documentation and examples to showcase features and make it easier to get started with the API". Cucumber solves this if you think reading tests is a good substitute for documentation and worked, comprehensive examples in other languages :-)

My experience with cucumber is that it made writing tests slow, and running tests slow, and debugging tests harder.

@BRMatt

Sorry if I've misunderstood the problem, but have you considered using a tool like api-blueprint to document the API, then use tools that test the API against the defined documentation?

It adds some semantics on top of Markdown for documenting requests/responses/schemas etc. and it's pretty readable, even when browsing via Github.

Admittedly it probably won't capture intricate scenarios as easily as a cuke scenario/bash script, but thought it might be worth bringing up.

@tef
tef commented

(nb I'm not against having documentation and examples which can be tested to check they work, but I don't think that is the same as having tests that serve as documentation and examples. c.f python's doctests and the problems with using them)

@steveklabnik

@BRMatt , I'm not opposed, but one of the tricky bits here is that as Balanced moves even further towards hypermedia, we want to move away from documenting exact URLs and HTTP methods, and focus more on workflows. I don't have the most experience with api-blueprint, but generally, these kinds of tools tend to focus on 'RESTish' style documentation rather than this kind, so that's something that's in here too.

@steveklabnik

@tef, I think your 'use case for cukes' is the same thing, just more cynical. ;)

I actually didn't even come up with cukes until the last minute, so maybe they are described as a fit for cukes, but it happened in the other direction.

I don't think new people will go read a bunch of cukes, but it would be nice to show as a response to this kind of question.

@mattwynne

@tef makes an important point that the demands of testing and documentation can come into conflict. The RSpec project have tried this, and found that the comprehensive test suite made for pretty boring user documentation. Users don't want to see every single edge case.

My advice in this situation is usually to push the edge cases down into unit tests. If you think there's still benefit in having that complete specification readable by everyone, consider writing two sets of cukes - one that provides a user manual, and another that provides your complete specification.

@Kosmas

@tef regarding the issue of speed with the tests, I think that is something that will change soon but @mattwynne will be able to confirm this.

I also believe that the cukes would help to have a quick understanding of how things are supposed to work, but in order to fully understand you would have to go further deep into the code.

@MattyO

The python equivalent to cucumber is called lettuce. It a little immature compared to cucumber. Its usefulness may be found in the use cases of interoperability with any current testing tools and libraries you might have or want to use in the future written in python.

Mainly wanted to let you know of its existence.

@BRMatt

@BRMatt , I'm not opposed, but one of the tricky bits here is that as Balanced moves even further towards hypermedia, we want to move away from documenting exact URLs and HTTP methods, and focus more on workflows. I don't have the most experience with api-blueprint, but generally, these kinds of tools tend to focus on 'RESTish' style documentation rather than this kind, so that's something that's in here too.

@steveklabnik Fair enough, those are reasonable points. I believe the maintainers of api-blueprint are trying to work in support for hypermedia relations, but I'm not sure how that fits in with your plans.

@mjallday
Owner

@steveklabnik do you see Balanced then implementing the underlying scenarios for the Gherkin in each client library to ensure they conform to a basic functionality (the spec)?

As I'm sitting here figuring out how we can implement the rev1.1 clients I really like the idea of a basic set of functional tests across all client libraries so we don't end up with the issues we had last round such as under-utliized client libraries having issues opened because they are missing esoteric functionality and we don't realise until someone actually tries to use it.

@mahmoudimus

Here are my thoughts:

I think Gherkin is the way to go here, definitely +1 on it.

First and foremost, there are many Gherkin/Ragel parsers in many different languages. The official client libraries that are supported are: Ruby, Python, PHP, Java, C# and Node.js -- all of which have Gherkin/Ragel runners. So this is largely aiding #4 as well and helps with the training wheels.

We attempted to something very similar to this in our documentation: https://github.com/balanced/balanced-docs/tree/master/clients -- maybe instead of having scenarios like this, we organize tests implemented in different languages and use those to feed into our API? Ben might have more thoughts on this topic.

This is definitely a step in the right direction and however, I'd like to compare it to something like http://apiary.io or RAML.

@steveklabnik

@mjallday maybe; I'm not sure how directly applicable they are, as the clients would need testing that they consume things correctly, and these are tests that stuff is produced correctly.

That said, we should integration test both together, at some point.

@steveklabnik

I just realized that I wrote a blog post a few years back that's directly relevant to this conversation http://blog.steveklabnik.com/posts/2011-12-20-write-better-cukes-with-the-rel-attribute

@JonRowe

What'd be great is a way to visualise the gherkin documentation online, with each step able to sho the raw response / differing language implementations...

@steveklabnik

@JonRowe , that's what http://relishapp.com is for :)

@JonRowe

It allows you to visualise the documentation, but it's not very non-developer friendly, I found @chrismdp's rack-usermanual idea very intriguing in this respect. It also doesn't currently offer a means to inspect what it actually does / provide sample responses.

@steveklabnik

Oh that is super cool!

@aslakhellesoy

@steveklabnik we'd be happy to set you up with a free https://cucumber.pro/ account once we go live. Cucumber Pro will replace Relish.

Regarding the idea to document the API with Cucumber. If I was a developer using the API, I would like to see scenarios like this:

Feature: Payment

  Scenario: tokenize
    Given some precondition here
    When I http POST to /cards with JSON body:
      """
      {
        "number": "4111111111111111",
        "expiration_month": "12",
        "expiration_year": 2016
      }
      """
    Then the response code should be 201 with:
      """
      {
        ... whatever the expected reply is...
      }
      """

This is more verbose than what you suggested, but I think this level of detail is needed for the audience you're targeting (developers using the API)

@steveklabnik

@aslakhellesoy thanks! Yeah, that makes sense; with something like this, I'm never 100% sure what the right level of abstraction is

@aslakhellesoy

To find the right level of abstraction, ask some stakeholders (in your case, users of the API):

  1. Does this make sense?
  2. Would you be able to do this yourself with this info?
  3. Do you need more info?
  4. Is anything puzzling you?

Looking at your feature I think most developers unfamiliar with your API would answer like this:

  1. Yes, I think so
  2. No. I have no idea what HTTP request to issue
  3. Yes, can you give me an example?
  4. I don't understand how the secret key is relevant. Can you explain? (that's what the scenario description is for)
Feature: feature name
  Scenario: scenario name
    Some descriptive text here is useful

    Given ...
@mahmoudimus

@aslakhellesoy that looks really good! :+1:

@andrewsardone

We've been using a similar setup at @nutshellcrm. I think it has been working pretty well, though I admit it's tough to find the right level of abstraction.

@chrismdp

@steveklabnik thanks. Just realised the Rack::Usermanual demo links were broken: should be fixed now.

@steveklabnik

@andrewsardone would you be interested in some help and / or collaboration to turn this into a gem, maybe?

@steveklabnik

@tef, does this style do anything to mitigate your concerns, or you still think it's a terrible idea?

@steveklabnik

@aslakhellesoy @mattwynne , one more question: We have scenarios that operate like this: https://github.com/balanced/balanced-api/blob/revision1/scenarios/cards/tokenize_without_secret.yml#L20-L23

This (kinda terrible because of the runner) test basically says "run the 'tokenize' scenario, and then make a request to the URI that's located at cards.href in its response."

This kind of thing is going to come up a lot as we move more and more towards hypermedia; "execute this thing, if it passes, execute this other thing, based on what comes back." These feel like two different tests to me, but GivenScenario was depreciated, so I'm not sure how to accomplish this kind of task.

Thoughts?

@mamund

I've thought about this, too. Another possible testing challenge is that you're building a client that could possibly travel several paths to a successful outcome. So it's not just if this then that but also something like start at A and IF you eventually get to Z that's a pass ELSE that's a fail

A likely situation for hypermedia, but not something I am aware that current testing tools can handle.

@matin
Owner

@steveklabnik Could you also add a scenario that debits/charges the card that you just tokenized?

One of the most painful parts of our current scenarios is referencing the result of a previous scenario. This becomes even worse if you want to use some sort of a fixture to avoid doing repetitive tasks like tokenizing a card or creating a customer for different scenarios.

@steveklabnik

Yup, that's the next one I'm working on. That's how I found out #431 (comment) , because they all chain off of each other.

@andrewsardone

@andrewsardone would you be interested in some help and / or collaboration to turn this into a gem, maybe?

Absolutely! It would be good to get a few minds together to see if there’s any place for a specific tool. Our current setup is very simple – it really just glues together lostisland/faraday & collectiveidea/json_spec within features/support/env.rb, but perhaps we should all focus on not reinventing the wheel whenever we try to spec out a new API. I think your and @balanced’s approach here is the right one, and hopefully more API builders will follow your lead.

I’ve created a blank slate umbrella project for the gem. If others think there’s value in trying to build a simple tool, open up some issues and let’s see what we need!

@matthewfl

This looks good so far. I think that the main thing that I am waiting to see is how referencing the result from other scenarios will work, but there seems to already be conversation on this front.

I haven't run this locally yet, so I am wondering what the error reporting will look like for cukes. With the current yaml based scenarios, it first tests to make sure that the scenarios themselves are internally consistent, which basically ensure that you are not referencing non existent scenarios and files. Once it has finished the first step it then tests the api, and if there are any differences it basically ends up printing out all the json and reporting that there is some error. I am not sure if this is the right way to go about it, but how can you detect errors in the scenarios vs errors in the api.
https://travis-ci.org/matthewfl/balanced-api#L2182

I know that you are trying to go in the direction of including json in the scenarios (as that is what the target audience cares about), but I am questions what is the right level of detail in the scenarios. Currently you are listing that I will get back a cards with is an array of something. ( https://github.com/balanced/balanced-api/blob/cukes/features/cards/tokenize.feature#L26-L36 ) In the yaml based scenarios we had try to divide up what was content of the response by using specifically json schema and the the scenario just contained specific details for that instance. ( https://github.com/balanced/balanced-api/blob/revision1/scenarios/cards/card_tokenization.yml#L50-L53 )

@steveklabnik

Here's one example if your cukes are off: https://travis-ci.org/balanced/balanced-api/builds/14911965 It's good to draw the distinctions between the two, you're right, and cucumber absolutely does. Here's an assertion failure:

Then I should get a 200 OK status code                                     # features/step_definitions/card_steps.rb:17                                        
      <200> expected but was                                                                                                                                       
      <401>. (MiniTest::Assertion)                                                                                                                                 
      ./features/step_definitions/card_steps.rb:18:in `/^I should get a (.+) status code$/'                                                                        
      features/cards/tokenize.feature:38:in `Then I should get a 200 OK status code'                                            

And I used the schema here because that's what the YAML version had done. You're right that that test would end up having different ones.

@steveklabnik

@andrewsardone awesome, I will end up checking this out. :+1:

@steveklabnik

I found http://stackoverflow.com/a/4902507 , which might be a reasonable answer: I forgot that the actual Given/When/Then are for communication only. I like the idea of just adding a space and then describing the second bit.

This won't help reduce duplication, but it will make some things easier.

@steveklabnik

I used the SO technique to add the second follow-on scenario: https://github.com/balanced/balanced-api/blob/19c11dfab1ef888ad13b4e205a402e352220fa73/features/cards/tokenize.feature#L37-L39

Two big problems:

  1. Our schemas claim to be version 4, but they're actually version 3. https://github.com/balanced/balanced-api/blob/revision1/tester/runner.py#L68-L73
  2. Our schemas aren't actually valid schemas. They use the required style of version 3, but the version 4 of everything else.

Using non-standard versions of these schemas make me :'(

@steveklabnik

I fixed the version 4 issue, still have to worry about relative URI resolution.

This test also now demonstrates two different ways of handling the schemas: inline and in a file. I think I like inline for shorter things, but file stuff for longer ones, rather than just choosing one or the other.

@steveklabnik

Okay! So now it's just failing because of a schema validation. For some reason, it properly de-references the relative schemas when I pass in a File object rather than just the parsed JSON: 0cf9ff5#diff-371d49fabfca7e84475425c67890d2f9R43

I pushed it up to Travis while it fails so you can see what a failure would look like: https://travis-ci.org/balanced/balanced-api/builds/15200929

Tomorrow morning I'm going to figure out why it's actually failing, and then we can start really cranking out the scenarios. :+1:

@mattwynne

@steveklabnik these are looking good. It's nice to have some real examples to help the discussion.

Are you still worried about chaining scenarios together, or is it working out for you just to have a longer scenario?

The danger with having fewer long scenarios is that the feedback you get when it fails can be crude (or even obscure other failures later in the scenario). Smaller scenarios generally give you better feedback when they fail, but you need to be pragmatic. It often makes sense to assert a few things in the same scenario as you're doing in that one.

When I want to have several scenarios that test different stages or paths through a workflow, I'll generally do it like this:

Scenario: Complete stage one
  Given I have asked for a loan
  When I fill out my name
  And I fill out my age
  And I submit the form
  Then I should be on stage 2

Scenario: Complete stage two
  Given I have completed stage 1
  When I fill out my address
  And I fill out my favourite colour
  And I submit the form
  Then I should be on stage 3

Scenario: Complete stage three
  Given I have completed stages 1 and 2
  When I fill out my weight
  ...etc.

Point being, you roll up the details of the previous scenario into a single step. In the step def for that higher-level step, you can either use steps to literally delegate to Gherkin, or (preferably) just directly call the necessary Ruby to get the job done.

@steveklabnik

@mattwynne I'd still prefer ultimately to not have longer scenarios: for example, creating a card will be used by basically every other scenario, and I'd prefer to organize them by use case rather than have the same things over and over in the background. Not to mention the duplication, as well as what you just pointed out with regards to feedback.

I think that your example is basically perfect, it's just slightly annoying that when I call steps I have to enumerate all of the steps, when I know I always want to run every step in a given scenario. Does that make sense?

I really appreciate everyone on this thread. :heart:

@mattwynne

Yeah, it's slightly annoying, but suck it up ;)

If it were me, I'd make sure I had a Ruby method called create_card and delegate to that in all those other scenarios. You only need one or two scenarios that explain how the card creation process works in detail. You might well find that those detailed scenarios actually do things you don't need to do in other scenarios that just need a card to exist.

@steveklabnik

Cool. Everyone, I added a rake task and formatter generously adapted from http://blog.josephwilk.net/ruby/cucumber-tags-and-continuous-integration-oh-my.html to handle CI, so I can work on these specs while we determine what exactly is up.

@steveklabnik steveklabnik Nuke everything :bomb:
Also, start over again with Cucumber, and import the fixture schemas
from before.
29c4da7
@steveklabnik

I am in a happy place with all of this now. @matthewfl, what do you think? Ready to give me a hand in porting other ones over?

@matin
Owner

@steveklabnik I can't open the first link in your last comment

@steveklabnik

@matin edited, that's what i get for copy/paste

@steveklabnik
@steveklabnik steveklabnik "Tokenize a card without a secret" works.
There's a teeny bit of extra infrastructure to get a token. I built it by
accident, but we'll need it eventually, so I just left it in.
0249dca
@steveklabnik

I added a checklist of scenarios that we have, so we can measure progress.

@matthewfl

All of the CRUD scenarios for all the resources are complete. It does not make since to get a single event in most cases as everything provides a events index that can be used to list all events, eg: /debits/:debit_id/events

For referencing components of other scenarios, there is now hydrate that will substitute in :name_of_something in url and json body that are getting posted. We are still not really using the power of hypermedia as we are currently doing stuff like /cards/:card_id/debits whenever we need to reference another portion of a response.

steveklabnik and others added some commits
@steveklabnik steveklabnik Set up rake tasks to make running features easier
Three things here:

1. I'm going to translate features over, and mark them as failing when
they are failing. I'll also tag them with gh-N to mark GitHub issue #N
that they go along with.
2. `bin/rake cruise` has been set up to run for CI. Basically, it runs all
the things that aren't failing, and makes sure they pass. Then, it runs
all thet hings that are supposed to be failing, and makes sure they
fail. If either of those are not true, it fails the build, and if
they both pass, they pass. This way we know when something starts
passing again, and to remove the failing tag.
3. `bin/rake focus` is added when developing new scenarios. You just tag
the scenario you're working on with `@focus`, and then `bin/rake focus`
will run just that one scenario.
999670a
@steveklabnik steveklabnik New feature: creating API keys.
There was no existing schemas for the API key response, so I made one.
e48738e
@steveklabnik steveklabnik New feature: retrieve API keys e494c51
@steveklabnik steveklabnik New feature: list all API keys 82780ea
@steveklabnik steveklabnik New feature: remove API keys c3e3f5b
@steveklabnik steveklabnik Refactor/reorganize steps for clarity.
It's important not to refactor steps _too_ much, but these were certainly
http steps and not card steps.
7a8e23a
@steveklabnik steveklabnik Update README. 65a81b8
@matthewfl matthewfl moving root_url and accept_headers into global variables 8397390
@matthewfl matthewfl adding card tokens schema 6f28333
@matthewfl matthewfl Refactor schema error handling, reintroduce failure
We should handle these errors better, and now that we are,
let's reintroduce the failing second half of the scenario.

I'm actually not entirely sure that flunk-ing with the message
is any different than just bubbling up the error, but at least
it's a real assertion failure error, rather than a JSON API
parsing error.
f4328d5
@matthewfl matthewfl basic create customer scenario 224b089
@matthewfl matthewfl micro library to contain interaction with the api 43349be
@matthewfl matthewfl New feature: Customer index c318fb3
@steveklabnik steveklabnik Adding technical documentation. 424c741
@steveklabnik steveklabnik clean up some whitespace 26bbee3
@steveklabnik steveklabnik New feature: get customer f385d4f
@steveklabnik steveklabnik 'fix' failing address spec.
This was failing, but for the wrong reasons. Let's fail for the right ones.
aeb357e
@steveklabnik steveklabnik Clean up some small formatting issues.
I missed these in my earlier cleanups.
c66f665
@steveklabnik steveklabnik New feature: remove a customer 4d05364
@steveklabnik steveklabnik New feature: create a callback 146c870
@steveklabnik steveklabnik New feature: get callback info 6b9aae4
@steveklabnik steveklabnik refactor to use the client
Thanks @matthewfl
767de1e
@matthewfl matthewfl adding hydrater 1e61843
@matthewfl matthewfl start changing over to hydrater bbb2eff
@matthewfl matthewfl feature: debiting a card 7caf6fd
@matthewfl matthewfl feature: Retrieving debits for a card e7eaf45
@steveklabnik steveklabnik New features: list/delete callbacks c88e3e7
@steveklabnik steveklabnik New feature: get a card 9a4cb93
@steveklabnik steveklabnik New features: list and remove cards 9a6603e
@steveklabnik steveklabnik client library refactor.
I took @matthewfl's work on the client library and fixed it up a bit.
There were some idiom issues, some conventions that we're followed,
and a couple of OOP things that _really_ bugged me.

I'm happy with this now, other than two things: we are doing OO terribly
with this whole response thing, and we should also implement DELETE.
35776fa
@steveklabnik steveklabnik Implement PUT in the client. 26878a3
@steveklabnik steveklabnik New feature: update card 647f631
@matthewfl matthewfl feature: debit a customer 7c2c8e6
@matthewfl matthewfl spacing is hard 492b2f8
@matthewfl matthewfl bank accounts feature failing as address is not coming back in response b6cfe2b
@matthewfl matthewfl credit bank account feature ca0e231
@matthewfl matthewfl basic crud for bank accounts 125f31b
@matthewfl matthewfl bank account debit and credit scenarios 45654ef
@matthewfl matthewfl bank account verification feature aaa5fc4
@matthewfl matthewfl finishing debit scenarios and forcing json schema formatting fe968e5
@matthewfl matthewfl refund features e52e3f9
@steveklabnik steveklabnik Fix broken callback scenario.
Thanks for setting me straight, @mjallday
ea7a736
@steveklabnik steveklabnik new feature: debit a card 72b5546
@steveklabnik steveklabnik Move features into rest
We'll keep these CRUD ones here, and do more intersting cases in the main folder.
24c22bb
@matthewfl matthewfl customer add card/bank account 6ecfbee
@matthewfl matthewfl credits features b7e3b68
@matthewfl matthewfl reversal features 07fe36d
@matthewfl matthewfl events schemas 8db133e
@matthewfl matthewfl events features 12689e8
@steveklabnik

Excellent. Every commit in this pull request up to 12689e8 passes all of the tests, so we know bisect will work.

I'm now going to tackle the non-crud examples.

steveklabnik and others added some commits
@steveklabnik steveklabnik Add an alias for requests using different language
The phrase "I POST to /foo" and "Make a GET request to /foo" both
sound 'correct.' So let's support both, depending on whichever
sounds better.
f06b356
@steveklabnik steveklabnik New feature: root resource bce3379
@steveklabnik steveklabnik New feature: the resources resource f21ca0d
@dclausen dclausen minor spelling updates cf4036b
@steveklabnik steveklabnik Better error messages on status code failure.
Basically, "Got a 409, expected a 201" isn't actually very helpful.
We can grab the error description out of the body and print it out,
and it's way more helpful.
d5c1a3e
@steveklabnik steveklabnik New feature: create a reversal.
This is probably the worst commit I've made in a while. I'm basically
not happy with any real part of it. But it works.

We use the url_template gem to handle url templating, but it isn't
expanding variable names correctly, so I'm doing that myself. With
gsub. Hopefully I can clean that up.

Furthermore, we need to sort of store an 'environment' of mappings
of ivars to the template names. Global-ish mutable state makes me
very uneasy...

@matthewfl, I want your opinion on how to not make this suck.
8e820bf
@steveklabnik steveklabnik New feature: failed reversal.
Issue #455 is tracking the failure.
967a193
@steveklabnik steveklabnik New feature: failed reversal on bank account.
Issue #456 is tracking the failure.
b89b6e3
@steveklabnik steveklabnik New feature: successful reversal ed9340d
@steveklabnik steveklabnik New feature: debit a card b2a5c7c
@steveklabnik steveklabnik make a distinction between having a card and a customer card.
We were doing it the wrong way before.
8325137
@steveklabnik steveklabnik New feature: debit a customer card 1627bab
@steveklabnik steveklabnik New feature: create a customer 602214e
@steveklabnik steveklabnik fix description around customer cards ef2eecb
@steveklabnik steveklabnik new feature: refunds create 71d9ae0
@steveklabnik steveklabnik Add HTTP PATCH to tiny client.
We need this for increasingly more scenarios
2a70aac
@steveklabnik steveklabnik Log secret and marketplace ID
This is needed for debugging when things go sideways.
3d1c8bf
@steveklabnik steveklabnik New feature: create a debit 27ed2d1
@steveklabnik steveklabnik New feature: refunds go to original funding source.
I made the last scenario mis-understanding the way that the YAML scenarios worked.

Often, they have duplicate stuff with new names just to make sure that things work.
That means I don't need to translate them 1-1, I have to figure out which
thing is actually important and just do that.
a9d699e
@steveklabnik steveklabnik New feature: create a failing refund
This is currently not working. #459 is tracking it.
44512a5
@steveklabnik steveklabnik New feature: debit a verified bank account e866535
@steveklabnik steveklabnik New feature: debits to unverified bank accounts fail.
Note that this is different than the old spec, which was incorrect.
40f037d
@steveklabnik steveklabnik New feature: debit a customer b6bd5a7
@steveklabnik steveklabnik New feature: fail to debit a customer
I don't know why this is failing. #460 is tracking it.
49ece83
@mjallday

creating the refund will make a new card

?

i'm not sure i grok that comment.

Currently, the steps are implemented in such a way that calling the step in the next line makes a new card, which clobbers @card_id. Yay global mutable state!

I want something to make this stuff better, but the perfect is the enemy of the good, so I'm just sucking it up right this moment.

@mjallday

are you supposed to be checking the refund went to the original card (as opposed to the debit)

You fetch the debit, and then assert on the body of the debit, see 3 lines down where it checks for the original card id.

@matthewfl

@steveklabnik @rserna2010

I am just quickly looking over the changes for the last few weeks, and I am noticing that there are a lot of scenarios that are marked @failing I am thinking that this is getting to the point of being a smell where we are allowing ourselves to write stuff that looks good, but doesn't actually function. I am thinking that we should go through and carefully review all the cases where we are making scenarios as failing and probably remove the failing tag all together and attempt to work out the minor issues in the api that originally provoked that tag.

In my quick browsing of the code here are two cases where @failing is being abused.
https://github.com/balanced/balanced-api/blob/cukes/features/credits.feature#L20-L30 (making a credit will not validate against the debit schema)
https://github.com/balanced/balanced-api/blob/cukes/features/customers.feature#L39-L45 (afaik we did not start supporting the ruby style #{})

@steveklabnik

@matthewfl yup, the idea was that @rserna2010 would be able to pitch in where he could to help keep moving this forward, since you were out. I told him that I'd handle making sure they actually worked, I just merged them in as @failing so it'd be simpler to knock them out.

When this gets merged there should be zero @failing.s

steveklabnik and others added some commits
@steveklabnik steveklabnik Some movement toward #467 7736c9e
@steveklabnik steveklabnik Passing: reverse a credit on a bank account.
I had the wrong fixture data, so it was creating a failed credit, rather than a succeeded one.
d0d02c1
@steveklabnik steveklabnik remove 'set the default funding source
In 1.1, only instruments can be credited, so a default funding source doesn't make sense.

See #460
59337e0
@steveklabnik steveklabnik Some movement on underwriting a customer
See #468 for more.
e11eda8
@steveklabnik steveklabnik Moving forward on 'create an order'
There's still a failure, unsure which part is right or wrong.
0536af7
@steveklabnik steveklabnik progress on checking escrow on orders.
Same delivery address issue
a68ddd8
@steveklabnik steveklabnik Add in debugging steps.
Previously, @matthewfl had added some logging stuff, this is a bit
heavier, but often useful.
f7a6c77
@steveklabnik steveklabnik Making some progress on 'checking escrow of order'
Erroring for the same reason the other Orders are.
fba6285
@steveklabnik steveklabnik Making progress on 'orders can't be credited more...
... than escrow balance.
5ee8012
@steveklabnik steveklabnik Making progress on 'Create a refund' 5d33e55
@steveklabnik steveklabnik Update interpolation syntax
I missed this one because the scenario fails before it hit this part.
But it's pretty obvious it needs fixed.
3e1b0b8
@steveklabnik steveklabnik Typo fix: should be an order not an error
The scenario failed before this, but I want to catch it now. Bady
copy/paste on my part
870fcdb
@steveklabnik steveklabnik Making progress on create a reversal 51c6a76
@steveklabnik steveklabnik fix interpolation syntax
let's just do this all at once
37043f4
@matthewfl matthewfl baby steps 4ce588a
@steveklabnik steveklabnik New feature: underwrite a customer
Closes #468
6083a08
@matthewfl matthewfl Merge remote-tracking branch 'origin/cukes' into cukes
Conflicts:
	features/customers.feature
6c29870
@steveklabnik steveklabnik making some movement on #471 585f67f
@steveklabnik steveklabnik Make progress on 'transactions inherit descriptions' 48ec874
@steveklabnik steveklabnik make some progress on crediting an unverified merchant.
#474 is tracking this particular failure.
48b6239
@steveklabnik steveklabnik Making progress on 'cancelling an order' 413775d
@steveklabnik steveklabnik Making progress on 'existing buyer makes a purchase with a new card'
#476 has the report
0afcceb
@steveklabnik

And with this latest commit, we now have 100% of the scenarios ported over.

There are still failures due to backend bugs, but we're almost there!

matthewfl and others added some commits
@matthewfl matthewfl some cleaning 37eb86b
@steveklabnik steveklabnik making progress on 'existing buyer makes a purchase with an existing …
…card.

Yet another #469 victim.
89da538
@steveklabnik steveklabnik Making progress on 'new buyer makes a purchase'
Another #469.
a4a51d7
@steveklabnik steveklabnik Progress on existing buyer makes a purchase with a new card.
Closes #476.
e7f8430
@rserna2010 rserna2010 Add credit card scenarios. 3cb8958
@steveklabnik steveklabnik making progress on 'adding a card to a customer' 8a3ffbf
@steveklabnik steveklabnik New feature: AVS postal code matches 0fb808b
@steveklabnik steveklabnik New feature: AVS postal code does not match 0c1d810
@steveklabnik steveklabnik New scenario: AVS Postal card is unsupported 7de997d
@steveklabnik steveklabnik Making progress on 'AVS postal code is unused.
#438 will fix this.
b96c9d9
@steveklabnik steveklabnik New feature: AVS street matches 5381186
@steveklabnik steveklabnik New feature: AVS street does not match 87aafe9
@steveklabnik steveklabnik Making progress on 'avs street match is null'
#438 will fix that.
ec98ab4
@steveklabnik steveklabnik Progress on 'detect a visa card brand'
#438 fixes
84903e0
@steveklabnik steveklabnik Make progress on 'detect a mastercard brand'
#438 fixes
9a972aa
@steveklabnik steveklabnik Make progress on 'detect an amex brand'
#438 fixes
f850887
@steveklabnik steveklabnik make progress on 'detect a discover card' ae18309
@steveklabnik steveklabnik Make some progress on retrieving card info
#480 is tracking this.
761af4e
@steveklabnik steveklabnik Progress on 'tokenizing a card'
#438 is the problem.
80b8eb3
@steveklabnik steveklabnik Making progress on luhn test failure e51d3a9
@steveklabnik steveklabnik new feature: unstore a card b47d8d0
@steveklabnik steveklabnik New feature: CVV matches 7924eb0
@steveklabnik steveklabnik progress on 'CVV doesn't match'
#438 fixes this.
4c454ed
@steveklabnik steveklabnik Progress on 'CVV is unsupported'
#438 strikes again
91847c7
@steveklabnik steveklabnik Making progress on CVV is unused
#438 again
0af3c98
@steveklabnik steveklabnik 'Add a card to a customer' now works 2c2a4b0
@steveklabnik steveklabnik Feature fixed: Debit a card 9a9e032
@steveklabnik steveklabnik Making progress on 'basic order flow' 2e7a0e6
@matthewfl matthewfl removing failing tag from card scenarios 9ba89de
@steveklabnik steveklabnik New feature: card metadata 84d3b4b
@steveklabnik steveklabnik New feature: fail to confirm a bank account verification e0ff12a
@steveklabnik steveklabnik progress on 'inferring bank names' 2044e73
@steveklabnik steveklabnik Make progress on 'add a bank account to a customer' 9513dd2
@matthewfl matthewfl cleaning orders features 5f9f1c7
@matthewfl matthewfl trying new link resolve system 1392564
@matthewfl matthewfl fix e4bc413
@matthewfl matthewfl another one working 2eec38b
@matthewfl matthewfl fixing bank account rest scenarios 2770646
@matthewfl matthewfl fixed 404 error 7d774b5
@matthewfl matthewfl canceling an order f802d28
@matthewfl matthewfl the end is nearing 4b13e43
@matthewfl matthewfl one more to go 5a1a65f
@matthewfl matthewfl I did it, its done, no more failing bf659a8
@matthewfl matthewfl some issues with test credit card numbers df27867
@steveklabnik steveklabnik properly tag luhn test failure.
Should be linked to #481
79fd4df
@steveklabnik steveklabnik Tagging CVV is unsupported failure. 8027255
@matthewfl matthewfl cvv card number scenarios working 79702b8
@steveklabnik steveklabnik Properly tagging cvv does not match a6fa221
@matthewfl matthewfl it is all working now f08a98d
@steveklabnik

I am currently re-running the tests over every commit to make sure that bisect works. Assuming that's successful, it's time to finally merge this sucker.

:shipit:

@matthewfl

@steveklabnik I just noticed that there are some scenarios that do not appear to be implemented, might we double check that we have everything

https://github.com/balanced/balanced-api/blob/revision1/scenarios/debits/create_new_card.yml

@steveklabnik

Okay! I have re-gone through and double checked. We were missing one or two. Let's get that CI building.

@steveklabnik steveklabnik merged commit 7c22c05 into revision1

1 check passed

Details default The Travis CI build passed
@mjallday
Owner

oh snap! :dancer:

@steveklabnik steveklabnik referenced this pull request in apiaryio/api-blueprint
Open

Hypermedia #13

@julianpistorius

@steveklabnik have you looked at 'behave' instead of Cucumber? It's Python and allegedly better than Lettuce. http://pythonhosted.org/behave/index.html

http://pythonhosted.org/behave/comparison.html

@steveklabnik

@julianpistorius I have not, no.

@steveklabnik steveklabnik deleted the cukes branch
@mjallday mjallday referenced this pull request in balanced/balanced-dashboard
Closed

Setup cucumber integration tests #1009

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.