A business scenario from a firm which transforms data ( experience from FinTech )

[nmondal]

Here is the file :

Feature:  data to next api by making rest calls

Scenario: read over tons of data to process and feed it next

Given url 'https://jsonplaceholder.typicode.com/posts'
When method get
Then status 200
# here I want to verify that 
# 1. The response is a non empty list. 
# 2. The response contains json structures where 
# all json structures are
# of the schema - { "userId": int , "id": int , "title": string , "body" : string }
# And then, we want to isolate text from body and title
# Pass it to next API call after filtering 
# Then do some more business processing
# Yes, we can call any other language, but then we want less code.
# How do I do it?
# Looks to us that there is a design flaw which made Karate not Turing Complete
# More over, there is no fixed iteration like SQL
# Let me know if that understanding is correct.

[ptrthomas]

I'll address the part of the question where you have provided a concrete example:

* def validator = 
"""
function(list) {
  var valid = true;
  var userValidator = function(user) {
    if (typeof user.userId  != 'number') valid = false;
    if (typeof user.id != 'number') valid = false;
    if (typeof user.body != 'string') valid = false;
  }  
  list.forEach(userValidator);
  return valid;
}
"""
Given url 'https://jsonplaceholder.typicode.com/posts'
When method get
Then status 200
And assert response.length > 0
And assert validator(response)

[nmondal]

I appreciate the response, but this is not entirely what we intended to test.

Observe the following :

1. We did not test for a non empty list. .length is a property of string too, but that is implementation issue not a problem with the design - you can fix it. But requires some more code - which should be trivial... Array.isArray() .

2. of the schema - { "userId": int , "id": int , "title": string , "body" : string }
   Here is why it gets incredibly hard. The naive current approach won't work - simply because schema means field1, field2,... must exists, and no other fields exist. That is a very crucial test.
   That test is entirely missing - and is hard to implement.
   https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/getOwnPropertyNames will help, but then, you need to compare field names - and ordering is not guranted .... isa nightmare.
   Being concrete - here is what we dont want to come up as an object - notice how current implementation passes it through:

{ "userId": int , "id": int , "title": string , "body" : string , "company" : "string"  }

3. Moving onto the tests:

if (typeof user.userId  != 'number') valid = false;
    if (typeof user.id != 'number') valid = false;

incredibly bad idea. if user.id was a floating point, that test would pass. Now one can argue that:
http://stackoverflow.com/questions/14636536/how-to-check-if-a-variable-is-an-integer-in-javascript
won't really have existed - if that check was in essence trivial. It is not. JavaScript was NOT build to do software testing - you need a different class of language to do so.
In fact I would argue, the framework must have these things thought about - and kept in place.

4. The tests are NOT lazy. That is, after first failure, once the variable switches to false, it should have come out of the loop. In a fin org, we had to deal with fast failure - we had to deal with 100,000 of such json objects or more.

This is an implementation bottleneck - but I find it perplexing that no such higher order functions are actually present within the framework.

5. Now that we have that code, if you want to change the schema of the JSON object to apply the same thing to another rest service, returning a different schema - we need to write all of these hard coded code back again, while in principle something can be written as :

// schema -> key value pair of field to type mapping 
// obj -> they are always a key value pair of field to value mapping  
is_valid_schema( obj, schema ){
  // the fields set are equal 
   obj.keySet == schema.keySet && 
// no field exists such that the value of the field is different from the type in schema 
    !exists ( obj.keySet ) where { !(obj[$.o] isa schema[$.o])  }
}
// check valid schema

and now, it is good. Now, in this form, we do not need to change the code ever, even if the schema or anything changes. These are next to impossible to do in languages which were not built for Test Automation. Neither Java, not Python, nor JavaScript cuts it. Groovy is close, but not close enough.

Next is the data splicing. From the response I need to arbitrarily project/select and create objects to pass it to the next rest call. That will also be another problem.

Again, let me fix the head on aspects here. :-)
The approach you have is cool, and I appreciate that and I have already shared it with all my colleagues, and I appreciate the hard work you have put - both to create and to open source - I know how incredibly hard these are and even we could not open source. So, I whole heartedly congratulate you.

But my worry is the path you are taking is treaded too many times already - and how to validate predicates should be the main focus - which I do not see much here. With the changes I am suggesting if you try to write libraries in JS or Java, you would see why it won't work.
You need to have a better class of language. That is precisely why I said - "it won't work."
People would have to write code - and code and more code w/o a DSL.

Hope this strikes a cord.
Thanks for your time to answer and post.

[ptrthomas]

@nmondal I'll leave it to readers to decide whether your points and concerns are valid.

[ptrthomas]

After some consideration, opened #3 specifically to solve for iteration better. Thanks @nmondal for the feedback.

[ptrthomas]

Oops correction, #4 is for the iteration syntax.

[ptrthomas]

Thanks to @nmondal for the awesome feedback. With the latest commit and the introduction of the match each syntax ( see #4 ), the solution has been simplified to:

Given url 'https://jsonplaceholder.typicode.com/posts'
When method get
Then status 200
And assert response.length > 0
And match each response == { userId: '#number', id: '#number', title: '#string', body: '#string' }