SCP2 Commands ignoring restriction to local

[justinatpnnl]

I created a generating search that pulls in events from a rest api. Everything works perfectly with SCP2 when testing on my dev box, but when deploying in a distributed environment I get an error from each of my indexers that it can't make a connection to the rest api (internet access blocked as intended).

Configurations tried:

• Added local=true to commands.conf
• Added local=true to @configuration() within the command itself (old method)
• Added type='streaming',distributed=False to @configuration() (scpv2 method according to docs)

All of these still gave me errors on my indexers. If I use splunk_server=local with my command from the web ui, it does make it stay local as expected. All of this seems to run contrary to the docs which state that "By default generating commands are configured to run locally in the streams pipeline".

A side note, there appears to be some incomplete commands.conf instructions for setting up distributed options in commands.conf:
http://docs.splunk.com/DocumentationStatic/PythonSDK/1.6.0/searchcommands.html#splunklib.searchcommands.GeneratingCommand

SCP 2

1. Add this configuration setting to your code
2. You are good to go; no need to restart Splunk

For now I have reverted to SCP1 and local=True is working as expected.

[liketic]

@justinatpnnl
Can you try type=stateful?

[justinatpnnl]

@likel
I wasn't sure where to add that since I don't see that option documented anywhere, but I was guessing that I should add that to the commands.conf settings. I did, but it didn't make a difference:

[mycommand]
filename = mycommand.py
chunked = true
type = stateful

[shakeelmohamed]

@justinatpnnl can you try putting type=stateful in the @ configuration() decorator. If that doesn't work can you send your app to devinfo@splunk.com and we'll take a look

[justinatpnnl]

Adding type='stateful' to @Configuration() results in an error.

ValueError: Illegal value: type='stateful'

I tried this again in the recently released 1.6.1, same issues. I will zip up a basic version of the app I created and send it to devinfo@splunk.com for testing if that helps. You can also simulate the issue I'm seeing by adding a requests call to an address that doesn't exist:

#!/usr/bin/env python

import sys, json, requests
from splunklib.searchcommands import dispatch, GeneratingCommand, Configuration, Option, validators
from time import time

@Configuration()
class opsgenieCommand(GeneratingCommand):

    def generate(self):
        r = requests.get("http://fake.address.isfake")
        alert = {'_time': time(), '_raw': 'Testing 123'}
        yield alert

dispatch(opsgenieCommand, sys.argv, sys.stdin, sys.stdout, __name__)

If run on a search head connected to multiple indexers, you will get the following error from each attached peer:

HTTPConnectionPool(host='fake.address.isfake', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10e9b40d0>

This simulates rather nicely the problem I'm having where:

1. The search is distributed to the indexers unexpectedly
2. The indexers do not have access to the external address called in the custom command

[datasearchninja]

I have just been runnind some tests, and 'distributed=False' seems to have the opposite effect to what would be expected. Setting this to True results in a command that is not distributed

[justinatpnnl]

@datasearchninja Confirmed! Setting my configuration this way did the trick:

@Configuration(distributed=True)

Nice find!

[shakeelmohamed]

The fix for this was merged to develop with #182