Skip to content

Analysing SPARKL event logs

miklosduma edited this page Aug 21, 2018 · 1 revision

Use the SPARKL CLI to read and analyse event logs.

Getting started

Use the sparkl listen command to listen to SPARKL events generated by all configurations in your user tree.

$ sparkl connect http://localhost:8000
$ sparkl login *******
$ sparkl listen

# Run any transaction to generate some events

svc:	Sequencer

name:	CheckPrime
svc:	Sequencer

	name:	n
	type:	integer
	value:	13

# Exit with ctrl-C

sparkl listen is the same as sparkl listen /. The / symbol stands for the root of your user tree.

Call sparkl listen on any configuration object - such as a service or operation, to see only the events that correspond to the object.

You can target objects by either ID or path.

Use sparkl object PATH to get the ID of an object.

$ sparkl listen Scratch/Primes_expr/Mix/Frontend/Log

# Only events generated by the Log operation appear
name:	Log
svc:	Backend

	name:	YES

	name:	n
	type:	integer
	value:	13

You can listen to:

  • Folders - sparkl listen Scratch/Primes_expr
    • To see all events generated inside that folder.
  • Operations - sparkl listen Scratch/Primes_expr/Mix/Frontend/CheckPrime
    • To see only the data events that correspond to the operation.
  • Fields - sparkl listen Scratch/Primes_expr/YES
    • To see all data events that involve the field either as an input or output field.
  • Services - sparkl listen Scratch/Primes_expr/Backend
    • To see service-up and service-down events. Listen to a sequencer type service to see transaction events. A transaction is made up of a transaction-start and end event and all the data events that come between.

Sample use case

We would like to monitor our crypto currency portfolio. We are interested in the following:

  • Current prices
  • Price change percentages
  • Errors

Consume operations for fine-grain logging

The configuration uses a set of consume operations for collecting:

  • Price changes - LogChange operation
  • Errors - LogErrors operation
  • Current price of bitcoin - SetPrice operation

We can subscribe to any of these consumes. For example, sparkl listen CryptoPortfolio/Mix/Impl/LogErrors will return only error events, should an error happen.

Tracking down events

Listening to LogErrors returns any expected error that might happen. However, it does not tell us where the error originates from. To track errors, we have to listen to the error field itself.

$ sparkl listen CryptoPortfolio/error

id:	L-YOL-HO9-1PE
name:	Error
svc:	Portfolio

	name:	error
	type:	term

Now we know that the error field is sent by the Error reply. In the mix we have several request operations that send a reply named Error. We can use the id of the reply to track down the offending operation.

$ sparkl object L-YOL-HO9-1PE
fields:	B-YOL-HO9-UA
id:	L-YOL-HO9-1PE
name:	Error
ref:	K-YOL-HO9-1NM
value:	0

The reply references the request it corresponds to. Let's lookup the id of the request.

$ sparkl object K-YOL-HO9-1NM
fields:	B-YOL-HO9-N6 B-YOL-HO9-TE
id:	K-YOL-HO9-1NM
name:	Delete
parent:	C-YOL-HO9-1HE
service:	F-YOL-HO9-JM
value:	0

It turns out the Delete request was responsible for the error. We can put a listener on this operation and start debugging.

Unexpected errors

It might happen that an unexpected error catches our configuration by surprise. The previously discussed errors were handled by our configuration. Though named Error or error, they were in fact data events. In the case of a bug or any unexpected error, SPARKL sends an error event to the solicit or request operation that caused the error. Think of them as something that gets through your try/catch statements.

We cannot know in advance which operation will cause trouble. It is worth putting a listener on the root folder of our configuration for debugging. This may result in a significant number of events. At the moment, the listen command does not have a filter for error events. You can easily introduce your own filters though.

Customising listeners

Addressing the previous use case, you want to put a listener on a configuration folder and act only on unexpected error events. To do that, you have to use the Python version of the SPARKL CLI library.

Using the listener in Python

You can use the Python code of the SPARKL CLI directly. If you invoke it as a Python function, the sparkl listen command returns a generator. It is easiest to use with a for loop.

> from sparkl_cli.main import sparkl
> sparkl('connect', 'http://localhost:8000')
> sparkl('login', '')
> events = sparkl('listen')
> for event in events: print(event)

Executing the above commands is identical to running sparkl listen after logging in.

Handling events

Each event's structure follows a set pattern.

  'attr': {
    'reply': '', 
    'service': ''}, 
  'content': [
      'attr': {
        'name': 'message', 
        'type': 'string', 
        'value': 'Hello, World!'}, 
      'tag': 'field'}], 
  'tag': 'reply'}

Due to this conventional pattern, it is easy to write filters against events.

from sparkl_cli.main import sparkl

if __name__ == '__main__':
  sparkl('connect', 'http://localhost:8000')
  sparkl('login', '')
  events = sparkl('listen', 'Scratch/CryptoPortfolio')

  for event in events:
    # Only print error events
    if event['tag'] == 'error':

The above sample will only print error events generated within the CryptoPortfolio folder. Besides using filters, you can also take actions on intercepting a particular event.

from sparkl_cli.main import sparkl

if __name__ == '__main__':
  path_to_mix = 'Scratch/CryptoPortfolio'
  sparkl('connect', 'http://localhost:8000')
  sparkl('login', '')
  events = sparkl('listen', path_to_mix)

  for event in events:
    if event['tag'] == 'error':

      # Stop all services in case of error
      sparkl('stop', path_to_mix)
      # And stop logging too

Instead of just printing an error event, the above code also stops all services running inside the CryptoPortfolio mix. This will stop all running transactions too. Calling the close method on the generator stops the events stream.

For a more complex example using multiple filters, an on-event and on-exit function, see our sample logging framework.

You can’t perform that action at this time.