ServicePulse doesn't receive all messages on "All messages"

[jenspettersson]

Describe the bug

Description

This morning we deployed a new feature that sends a new command to one of our endpoints. When I wanted to check ServicePulse to see that the expected commands were sent I missed two commands and got a bit worried, they MUST have been sent. Then I started my old ServiceInsight and did the same search, and there they where!

I started investigating and found more types of messages that where not being searchable in the "All messages" search. I found one event that triggered a long running process and I could find all the messages within that long running process, but not the event that started the process. It was visible if I looked at the Sequence Diagram on one of the messages in the process so I tried to search on the specific message id but even then I didn't find that message in the search result.

Expected behavior

All messages should be returned when searching in "All messages"

Actual behavior

All messages doesn't seem to be returned when searching...

Versions

ServicePulse: v2.3.2
Service Control: v6.7.6
SC Monitoring: v6.7.6

Please list the version of the relevant packages or applications in which the bug exists.

Steps to reproduce

N/A

Relevant log output

Additional Information

Workarounds

Search using Service Insight

Possible solutions

Additional information

[DavidBoike]

Hi @jenspettersson,

I would love to gather some more information. Primarily, both these apps should be using (roughly) the same API calls to ServiceControl to get their data, so I'd like to see if there are any differences. I'm hoping that this issue is "stable" and you can get each app to show the same discrepancy as you could before.

In ServiceInsight, go to the Logs tab at the bottom, and if you refresh the message list, it should give information for going to the API for the data refresh.

In ServicePulse, you'll have to go into the browser developer tools and go to the Network tab and look for a similar URL.

So then, what are the two URLs used by the two tools, and are they different? And then, if they're different, if you look at the returned JSON, is that where the data discrepancy lies? Or does the data look the same (or maybe they are truly the same URL) in which case that would lead me to believe there's some filtering or processing going on in one tool or the other that's different from its cousin.

If you need to submit confidential information to us, I'd encourage you to open a full support ticket at https://customers.particular.net

[jenspettersson]

Hi @DavidBoike,

The URLs are indeed different.

From Service Insight

http://localhost:33333/api/messages/search?include_system_messages=False&q=VerifyStartedSignCompleted&per_page=50

With the following headers:

[
  "Content-Encoding=",
  "Transfer-Encoding=chunked",
  "Vary=Accept,Accept-Encoding",
  "Total-Count=230",
  "Link=<messages/search?include_system_messages=False&q=VerifyStartedSignCompleted&per_page=50&page=3>; rel=\"last\",<messages/search?include_system_messages=False&q=VerifyStartedSignCompleted&per_page=50&page=2>; rel=\"next\"",
  "X-Particular-Version=6.7.6",
  "Cache-Control=private, max-age=0",
  "Content-Type=application/json; charset=utf-8",
  "Date=Mon, 24 Nov 2025 07:35:10 GMT",
  "ETag=f1e7e194-0cc7-cc98-2c00-91c9a3668c6b",
  "Server=Kestrel"
]

That query returns Total-Count: 230 messages.

From Service Pulse

http://localhost:33333/api/messages2/?endpoint_name=&from=&to=&q=VerifyStartedSignCompleted&page_size=100&sort=time_sent&direction=desc

With the following headerS:

Content-Type: application/json; charset=utf-8
Date: Mon, 24 Nov 2025 07:36:36 GMT
Server: Kestrel
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: ETag,Last-Modified,Link,Total-Count,X-Particular-Version,Content-Disposition
Cache-Control: private, max-age=0
Content-Encoding: br
Transfer-Encoding: chunked
Vary: Accept
Vary: Accept-Encoding
Total-Count: 114
X-Particular-Version: 6.7.6

That query returns Total-Count: 114messages.

I can't find any duplicates in the first Serviceinsight query, all the messag ids are unique, so the Service Pulse query is missing quite a lot of messages.

[DavidBoike]

Interesting, I didn't actually know that the two apps were using different APIs (/api/messages/search vs. /api/messages2/ to search messages. It's going to take me some time to trace through the differences in the API, but one thing I do notice is that the ServicePulse one has parameters for date filtering which are unset, but I could see how there could potentially be a bug where "no filter" still filters somehow. Again, will take some time to trace through, but if in ServicePulse you set the date filter to something sufficiently large to encompass all the messages ServiceInsight is showing you, does that change what you see in ServicePulse?

[DavidBoike]

I've been able to trace down all this code, and although I've traced the differences between the two query paths, they're mostly to do with the date range query that ServicePulse offers which ServiceInsight doesn't have. And with the (admittedly primitive) smoke testing messages I'm using, I can't force any different results.

@jenspettersson could you send us (I would advise a support case) the JSON (as returned by the ServiceControl URL) for a message that is returned by both ServiceInsight and ServicePulse (preferably via both APIs), as well as the JSON that is returned by a message that only shows up in ServiceInsight?

Also, I don't think this is the case, but I want to confirm that you have only one Audit instance, correct?