Json parsing error thrown while running the service consumer test #475

sushantchoudhary · 2017-06-28T04:09:04Z

Hey guys,

We have a pact-jvm(pact-jvm-consumer-junit_2.11:3.4.1) setup running the junit tests to generate the pact file.Recently we have started getting this error while running the tests. Stacktrace doesnt give a lot of info but looks like some discrepancy in reading the Pact file. Not sure if its how I am generating the response causing the issue.

groovy.json.JsonException: Unable to determine the current character, it is not a string, number, array, or object

The current character read is '?' with an int value of 0
Unable to determine the current character, it is not a string, number, array, or object
line number 1
index number 255
????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
...............................................................................................................................................................................................................................................................^
	at groovy.json.internal.JsonParserCharArray.decodeValueInternal(JsonParserCharArray.java:206)
	at groovy.json.internal.JsonParserCharArray.decodeValue(JsonParserCharArray.java:157)
	at groovy.json.internal.JsonParserCharArray.decodeFromChars(JsonParserCharArray.java:46)
	at groovy.json.internal.JsonParserCharArray.parse(JsonParserCharArray.java:384)
	at groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:128)
	at groovy.json.internal.BaseJsonParser.parse(BaseJsonParser.java:151)
	at groovy.json.JsonSlurper.parseFile(JsonSlurper.java:365)
	at groovy.json.JsonSlurper.parse(JsonSlurper.java:348)
	at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
	at au.com.dius.pact.model.PactReader.loadFile(PactReader.groovy:159)
	at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
	at au.com.dius.pact.model.PactReader.loadPact(PactReader.groovy:26)
	at au.com.dius.pact.model.PactReader.loadPact(PactReader.groovy)
	at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:232)
	at au.com.dius.pact.model.BasePact.write(BasePact.groovy:109)
	at au.com.dius.pact.consumer.BaseMockServer.runAndWritePact(MockHttpServer.kt:147)
	at au.com.dius.pact.consumer.ConsumerPactRunnerKt.runConsumerTest(ConsumerPactRunner.kt:13)
	at au.com.dius.pact.consumer.BaseProviderRule.runPactTest(BaseProviderRule.java:148)
	at au.com.dius.pact.consumer.BaseProviderRule.access$100(BaseProviderRule.java:21)
	at au.com.dius.pact.consumer.BaseProviderRule$1.evaluate(BaseProviderRule.java:76)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:114)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:57)
	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:66)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:109)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:377)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
	at org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

This is one of the interaction setup where we are getting this error,

    @Pact(consumer = CONSUMER_ID)
    public RequestResponsePact getBoardsForMobile(PactDslWithProvider builder) {
        return builder
                .given("Project with boards")
                .uponReceiving("GET all boards")
                .matchPath(BOARDS_API_PATTERN, BOARDS_API_EXAMPLE)
                .method("GET")
                .query("maxResults=10")
                .willRespondWith()
                .status(HttpStatus.SC_OK)
                .headers(ImmutableMap.of("Content-Type", "application/json"))
                .body(new PactDslJsonBody()
                        .minArrayLike("values", 1)
                        .id()
                        .stringMatcher("type", "CORE")
                        .stringMatcher("name", "Business")
                        .stringMatcher("moduleKey", CORE_MODULE_EXAMPLE)
                        .closeObject()
                        .object()
                        .id()
                        .stringMatcher("type", "SCRUM")
                        .stringMatcher("name", "DEMO board")
                        .stringMatcher("moduleKey", AGILE_MODULE_EXAMPLE)
                        .closeObject()
                        .closeArray()
                        .asBody())
                .toPact();
    }

And this is how the json from pact file looks like,

mobile-rest-plugin will respond with:

{
  "status": 200,
  "headers": {
    "Content-Type": "application/json"
  },
  "body": {
    "values": [
      {
        "name": "Business",
        "id": 8124081232,
        "type": "CORE",
        "moduleKey": "core-mobile-board-service"
      },
      {
        "name": "DEMO board",
        "id": 4597649782,
        "type": "SCRUM",
        "moduleKey": "agile-mobile-board-service"
      }
    ]
  }
}

The text was updated successfully, but these errors were encountered:

uglyog · 2017-06-29T00:37:14Z

The JSON parser is failing to parse the pact file. Looks like it might be either a corrupt file or the file has the wrong content type.

In what environment are the tests running? (Linux/Windows etc.)

sushantchoudhary · 2017-06-29T00:41:52Z

Cool, I will verify the content-type, is there a way to validate corrupt pact file?

uglyog · 2017-06-29T00:44:54Z

Open it in a text editor.

sushantchoudhary · 2017-06-29T00:52:02Z

cool, on it. thanks @uglyog

sushantchoudhary · 2017-06-29T07:07:53Z

Hey @uglyog , I looked at content-type values and also verified the pact json file but cant zero in on anything potentially causing this error. Any other element which might be causing it? Btw this surfaces only while running the test with gradle test , running it from IDE doesnt complain. This is my complete pact json file for reference,

{
    "provider": {
        "name": "mobile-rest-plugin"
    },
    "consumer": {
        "name": "jira-android"
    },
    "interactions": [
        {
            "description": "GET agile board",
            "request": {
                "method": "GET",
                "path": "/boards/1.0/board/1",
                "query": "markAsViewed=true&moduleKey=agile-mobile-board-service"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": {
                    "subscribed": "false",
                    "name": "DEMO board",
                    "id": 1,
                    "type": "SCRUM"
                },
                "matchingRules": {
                    "$.body.type": {
                        "regex": "SCRUM",
                        "match": "regex"
                    },
                    "$.body.id": {
                        "match": "integer"
                    },
                    "$.body.subscribed": {
                        "regex": "false",
                        "match": "regex"
                    },
                    "$.body.name": {
                        "regex": "DEMO board",
                        "match": "regex"
                    }
                }
            },
            "providerState": "JIRA project with agile board"
        },
        {
            "description": "GET all boards",
            "request": {
                "method": "GET",
                "path": "/boards/1.0/recent",
                "query": "maxResults=10"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": {
                    "values": [
                        {
                            "name": "Business",
                            "id": 7243125415,
                            "type": "CORE",
                            "moduleKey": "core-mobile-board-service"
                        },
                        {
                            "name": "DEMO board",
                            "id": 5636926176,
                            "type": "SCRUM",
                            "moduleKey": "agile-mobile-board-service"
                        }
                    ]
                },
                "matchingRules": {
                    "$.body.values[*].type": {
                        "regex": "SCRUM",
                        "match": "regex"
                    },
                    "$.body.values[*].id": {
                        "match": "type"
                    },
                    "$.body.values[*].name": {
                        "regex": "DEMO board",
                        "match": "regex"
                    },
                    "$.body.values[*].moduleKey": {
                        "regex": "agile-mobile-board-service",
                        "match": "regex"
                    },
                    "$.body.values": {
                        "min": 1,
                        "match": "type"
                    }
                }
            },
            "providerState": "JIRA project with boards"
        },
        {
            "description": "GET subscription setting",
            "request": {
                "method": "GET",
                "path": "/cards/1.0/events/subscription"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": {
                    "watcher": "false",
                    "boards": [
                        
                    ],
                    "assignee": "false",
                    "mentioned": "false",
                    "reporter ": "false"
                }
            },
            "providerState": "User can fetch subscription setting"
        },
        {
            "description": "GET board search",
            "request": {
                "method": "GET",
                "path": "/boards/1.0/search",
                "query": "q=board"
            },
            "response": {
                "status": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": [
                    {
                        "name": "EL Board",
                        "id": 4622810293,
                        "type": "SCRUM",
                        "moduleKey": "agile-mobile-board-service"
                    },
                    {
                        "name": "DEMO board",
                        "id": 8283190000,
                        "type": "SCRUM",
                        "moduleKey": "agile-mobile-board-service"
                    }
                ],
                "matchingRules": {
                    "$.body[*].type": {
                        "regex": "SCRUM",
                        "match": "regex"
                    },
                    "$.body": {
                        "min": 0,
                        "match": "type"
                    },
                    "$.body[*].name": {
                        "regex": "DEMO board",
                        "match": "regex"
                    },
                    "$.body[*].id": {
                        "match": "type"
                    },
                    "$.body[*].moduleKey": {
                        "regex": "agile-mobile-board-service",
                        "match": "regex"
                    }
                }
            },
            "providerState": "User can search for a board"
        }
    ],
    "metadata": {
        "pact-specification": {
            "version": "2.0.0"
        },
        "pact-jvm": {
            "version": "3.4.1"
        }
    }
}

sushantchoudhary · 2017-06-29T07:08:29Z

Also , the tests are running on mac (local) and Linux (CI).

BenSayers · 2017-06-29T08:01:31Z

Assuming that a previous test was generating a Pact file that the current test could not read I started commenting things out and eventually discovered that when we replace the matchPath method call:

return builder
                .given("User can fetch subscription setting")
                .uponReceiving("GET subscription setting")
                .matchPath("(/!?rest)?/cards/1.0/events/subscription", "rest/cards/1.0/events/subscription")
                .method("GET")
                .willRespondWith()
                .status(HttpStatus.SC_OK).headers(ImmutableMap.of("Content-Type", "application/json"))
                .body(GSON.toJson(response))
                .toPact();

with the path method call:

return builder
                .given("User can fetch subscription setting")
                .uponReceiving("GET subscription setting")
                .path("rest/cards/1.0/events/subscription")
                .method("GET")
                .willRespondWith()
                .status(HttpStatus.SC_OK).headers(ImmutableMap.of("Content-Type", "application/json"))
                .body(GSON.toJson(response))
                .toPact();

it fixes the problem.

This pact mock configuration is part of a test that runs and passes prior to the test that fails. I've just tried to reproduce the problem in a simpler example and was unable to get it to fail so I'm not too sure what exactly is tripping up the parser.

I hope this helps you guys narrow in on the root cause problem.

sushantchoudhary · 2017-07-02T23:53:20Z

Hey @uglyog , I tried using pact DSL directly for the consumer tests in hope that it will perhaps fix how pact file is read/written and also applied Ben's approach for path matcher but realized it doesn't makes a difference and test still fails with the same JSON parsing error. Please suggest if there is any other data point we can work against?

uglyog · 2017-07-03T01:10:17Z

From the stack trace, it looks like the failure happens when the test is successful, and pact-jvm is trying to merge the generated pact file with the one that already exists on disk. It is the one on disk that is causing the issue. Could you run a clean before the tests?

sushantchoudhary · 2017-07-03T01:54:21Z

yeah I did run gradle clean before every test run so that there is no pact file on disk.

Another observation, I have 3 classes and 4 unit tests and occasionally one test is not executed and the gradle test succeeds without any error. However, on the subsequent run all the 4 tests run and one of them fails(with reported error) while writing to the existing pact file (generated from previous tests) . Not sure why 😕

On a hunch that since gradle test run all the test-* tasks in my Android project, test classes were included in more than one task and causing the dirty file state, hence I excluded the consumer test package from all but one test-* task but unfortunately that didn't resolve the issue either 😞 .
Btw the test runs and generates pact file successfully if we run the test from IDE. Its the gradle test where it trips.

uglyog · 2017-07-03T04:28:41Z

Ok, we're getting closer. The different between running things in IntelliJ and Gradle is that Gradle could run tests concurrently in different threads. I'm assuming you're using the Android Gradle plugin? Are you're tests running as unit tests or instrumented tests (under androidTest)?

sushantchoudhary · 2017-07-03T05:57:37Z

Thats correct, they are unit tests ( not the instrumented tests) . And yes we are using Android gradle plugin to run the tests.

sushantchoudhary · 2017-07-03T06:16:09Z

Also, after making all the above changes, I think it has come to a point where all the tests are running successfully but the pact file doesn't capture all the interactions (2/4). Guess its just that I am not running into concurrency scenario now. Any hints?

uglyog · 2017-07-06T00:30:58Z

Can you check that the four interactions have a unique description. They may be overwriting the older ones (the provider state and description should be unique).

I'm going to add a file system lock to the pact writing code to protect against this type of issue.

sushantchoudhary · 2017-07-06T22:36:48Z

Thats right they are not always unique. This issue is surfacing because we were forking 4 test worker process for running our unit test using Junit runner. Gradle Test task provides a system property maxParallelForks to control the no of test process to execute in parallel.As a workaround, I have excluded the Pact test classes using the testOptions and running them as a test task with single test process(default is maxParallelForks=1) .

In module gradle,

    testOptions {
        unitTests {
            returnDefaultValues = true
            all {
                if (it.name != 'testNightlyDebugUnitTest') {
                    exclude 'com/atlassian/android/jira/core/contracts/**'
                }
            }
        }

    }

In root gradle,

    tasks.withType(Test).whenTaskAdded { testTask ->
        if (!testTask.name == 'testNightlyDebugUnitTest') {
            testTask.maxParallelForks config.testForks  // 4
            testTask.testLogging config.testOptions
        }

    }

Looks a bit clunky though 😞

uglyog · 2017-07-06T22:44:08Z

I definitely need to put that synchronisation check in before writing the pact file.

… with an existing one on the file system #475

uglyog · 2017-07-12T01:44:34Z

Version 3.5.2 has been released with synchronisation and file locking on the pact file.

… with an existing one on the file system #475 Conflicts: pact-jvm-model/src/main/groovy/au/com/dius/pact/model/BasePact.groovy pact-jvm-model/src/main/kotlin/au/com/dius/pact/model/Pact.kt

Conflicts: pact-jvm-consumer-junit/src/test/java/au/com/dius/pact/consumer/Defect464Test.java

uglyog added the bug Indicates an unexpected problem or unintended behavior label Jul 9, 2017

uglyog pushed a commit that referenced this issue Jul 9, 2017

Add a synchanisation block and file system lock around merging a pact…

262db0f

… with an existing one on the file system #475

uglyog pushed a commit that referenced this issue Jul 10, 2017

Updated the file system lock code to work on windows #475

e66db1d

uglyog pushed a commit that referenced this issue Aug 16, 2017

Updated the file system lock code to work on windows #475

fe7c1f7

Conflicts: pact-jvm-consumer-junit/src/test/java/au/com/dius/pact/consumer/Defect464Test.java

uglyog closed this as completed Feb 22, 2020

bkosawa mentioned this issue Mar 30, 2022

Random au.com.dius.pact.core.support.json.JsonException when running tasks in parallel #1528

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Json parsing error thrown while running the service consumer test #475

Json parsing error thrown while running the service consumer test #475

sushantchoudhary commented Jun 28, 2017 •

edited

uglyog commented Jun 29, 2017 •

edited

sushantchoudhary commented Jun 29, 2017

uglyog commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

BenSayers commented Jun 29, 2017 •

edited

sushantchoudhary commented Jul 2, 2017

uglyog commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

uglyog commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

uglyog commented Jul 6, 2017

sushantchoudhary commented Jul 6, 2017

uglyog commented Jul 6, 2017

uglyog commented Jul 12, 2017

Json parsing error thrown while running the service consumer test #475

Json parsing error thrown while running the service consumer test #475

Comments

sushantchoudhary commented Jun 28, 2017 • edited

uglyog commented Jun 29, 2017 • edited

sushantchoudhary commented Jun 29, 2017

uglyog commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

sushantchoudhary commented Jun 29, 2017

BenSayers commented Jun 29, 2017 • edited

sushantchoudhary commented Jul 2, 2017

uglyog commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

uglyog commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

sushantchoudhary commented Jul 3, 2017

uglyog commented Jul 6, 2017

sushantchoudhary commented Jul 6, 2017

uglyog commented Jul 6, 2017

uglyog commented Jul 12, 2017

sushantchoudhary commented Jun 28, 2017 •

edited

uglyog commented Jun 29, 2017 •

edited

BenSayers commented Jun 29, 2017 •

edited