Assertion errors given to the user only contain the string representation of values

[pubudu91]

When an assertion error occurs, it only uses the string representations of the values to communicate the error to the user. However, if the string representation is the same, but the types are different, the assertion still fails, but the user will be confused as to why the assertion failed when the two values are seemingly the same. Consider the following sample error which was a result of a type mismatch.

error: Assertion Failed!: expected '{name:"John Doe", age:25, address:{city:"Colombo", country:"Sri Lanka"}}' but found '{name:"John Doe", age:25, address:{city:"Colombo", country:"Sri Lanka"}}' {}
	    	at ballerina/test:createBallerinaError(assert.bal:42)
	    	ballerina/test:assertEquals(assert.bal:76)
	    	ballerina-examples/anonymous-records:0.0.1:testFunc(tests/anonymous_records_test.bal:19)

[Dilhasha]

Following is how the assertion comparison is done in JUnit5.

If there is a type mismatch, the errors are shown as follows.

• String "1" vs int 1
  expected: java.lang.String@28feb3fa<1> but was: java.lang.Integer@675d3402<1>

• Json vs String
  expected: java.lang.String@5025a98f<{"name":"Dilhasha"}> but was: com.google.gson.JsonObject@49993335<{"name":"Dilhasha"}>

When the types are matched, but the values are different, the errors are shown as follows.

• expected: <8> but was: <7>

In the following 2 examples, only the type of the property 'age' is different.

• expected: <{"firstName":"Sergey","age":18}> but was: <{"firstName":"Sergey","age":"18"}>

• expected: <{"firstName":"Sergey","lastName":"Harvard","age":18,"address":"879, Colombo Road","school":"primary","telephone":"+96738373933","occupation":"SSE","favouriteColor":"Green"}> but was: <{"firstName":"Sergey","lastName":"Harvard","age":"18","address":"879, Colombo Road","school":"primary","telephone":"+96738373933","occupation":"SSE","favouriteColor":"Green"}>

[Dilhasha]

Following is how the assertion comparison is done in GoLang.

If there is a type mismatch, the errors are shown as follows.

• string "1" vs int 1

    	Error:      	Not equal:
    	            	expected: int(1)
    	            	actual  : string("1")
  

• Json vs String

    	Error:      	Not equal: 
    	            	expected: []uint8([]byte{0x7b, 0x22, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x44, 0x69, 0x6c, 0x68, 0x61, 0x73, 0x68, 0x61, 0x22, 0x7d})
    	            	actual  : string("{\"Name\":\"Dilhasha\"}")
  

When the types are matched, but the values are different, the errors are shown as follows.

•      	Error:      	Not equal: 
    	            	expected: 8
    	            	actual  : 7
  

•      	Error:      	Not equal: 
    	            	expected: "Hello"
    	            	actual  : "New"
    	            	
    	            	Diff:
    	            	--- Expected
    	            	+++ Actual
    	            	@@ -1 +1 @@
    	            	-Hello
    	            	+New
  

•     	Error:      	Not equal: 
    	            	expected: main.Person{Name:"Dilhasha", Age:18}
    	            	actual  : main.Person{Name:"Dilhasha", Age:20}
    	            	
    	            	Diff:
    	            	--- Expected
    	            	+++ Actual
    	            	@@ -2,3 +2,3 @@
    	            	  Name: (string) (len=8) "Dilhasha",
    	            	- Age: (int) 18
    	            	+ Age: (int) 20
    	            	 }
  

• Two structs with different attributes

   	Error:      	Not equal: 
   	            	expected: []byte{0x7b, 0x22, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x44, 0x69, 0x6c, 0x68, 0x61, 0x73, 0x68, 0x61, 0x22, 0x7d}
   	            	actual  : []byte{0x7b, 0x22, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x3a, 0x22, 0x44, 0x69, 0x6c, 0x68, 0x61, 0x73, 0x68, 0x61, 0x22, 0x2c, 0x22, 0x41, 0x67, 0x65, 0x22, 0x3a, 0x31, 0x38, 0x7d}
   	            	
   	            	Diff:
   	            	--- Expected
   	            	+++ Actual
   	            	@@ -1,4 +1,4 @@
   	            	-([]uint8) (len=19) {
   	            	+([]uint8) (len=28) {
   	            	  00000000  7b 22 4e 61 6d 65 22 3a  22 44 69 6c 68 61 73 68  |{"Name":"Dilhash|
   	            	- 00000010  61 22 7d                                          |a"}|
   	            	+ 00000010  61 22 2c 22 41 67 65 22  3a 31 38 7d              |a","Age":18}|
   	            	 }
  

The function is as follows.

func TestJsonComparison(t *testing.T) {
    type Person struct {
      Name    string
    }
    type PersonNew struct {
      Name    string
      Age int
    }
    person1 := Person{ Name: "Dilhasha",}
    person2 := PersonNew{ Name: "Dilhasha", Age: 18,}
    var jsonData,jsonData2 []byte
    jsonData, err := json.Marshal(person1)
    if err != nil {
        log.Println(err)
    }
    jsonData2, err2 := json.Marshal(person2)
    if err2 != nil {
        log.Println(err2)
    }
    assert.Equal(t, jsonData, jsonData2, "")
}

[Dilhasha]

@hevayo

IMO the diff based approach used in GoLang is much useful compared to the approach in JUnit5. WDYT?

[Dilhasha]

Following is how the assertion comparison is done in TestNG. Same limitation of not showing the type mismatch as mentioned in this issue is present in TestNG too.

• String "1" vs int 1
  expected [1] but found [1]

• Json vs String
  expected [{"name":"Dilhasha"}] but found [{"name":"Dilhasha"}]

[pubudu91]

I agree, the diff approach leaves no room for guessing what actually went wrong. But that would only work for anydata types right? What about behavioural types such as objects, functions, typedescs etc.?

[Dilhasha]

@pubudu91
Thanks for the feedback.
We can diff the behavioral types such as objects as long as they can be broken down to consist of anydata types. For example, check the below struct assertion in GoLang. We can use a similar approach.

            Error:          Not equal: 
                            expected: main.FruitBasket{Name:"Standard", Fruit:[]string{"Apple", "Banana", "Orange"}, Id:999, private:"Second-rate"}
                            actual  : main.FruitBasket{Name:"StandardNew", Fruit:[]string{"Grapes", "Bananas", "Orange"}, Id:998, private:"Second-rate"}
                            
                            Diff:
                            --- Expected
                            +++ Actual
                            @@ -1,9 +1,9 @@
                             (main.FruitBasket) {
                            - Name: (string) (len=8) "Standard",
                            + Name: (string) (len=11) "StandardNew",
                              Fruit: ([]string) (len=3) {
                            -  (string) (len=5) "Apple",
                            -  (string) (len=6) "Banana",
                            +  (string) (len=6) "Grapes",
                            +  (string) (len=7) "Bananas",
                               (string) (len=6) "Orange"
                              },
                            - Id: (int64) 999,
                            + Id: (int64) 998,
                              private: (string) (len=11) "Second-rate"

Appreciate if you can share some examples where this might not be possible in the context of Ballerina. So, that I can understand it clearly.

[Dilhasha]

Now that "assertEquals" function accepts only anydata type values, we can show the diff when there is an assertion failure.

[Dilhasha]

Assertion failure can be handled in following 2 parts.

1. If assert fails and type mismatch, we can show the type and display the string representation as is.
2. If there is no type mismatch, we can check on ways to show the diff

[Dilhasha]

Assertion diff for other data types will be addressed in #26204