Skip to content

Commit

Permalink
Support list unroll to hash function
Browse files Browse the repository at this point in the history
- Added an initial implementation of the unroll(hash,'key','value') function
  for JMESPath.  Still not sure what it should be called, but the logic should
  word.  Closes #5.
- Add an online appinspect checking script.
  • Loading branch information
lowell80 committed Nov 9, 2018
1 parent e57eb27 commit 411a197
Show file tree
Hide file tree
Showing 3 changed files with 121 additions and 0 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ venv
.latest_release
build
dist
app_inspect*.json

# Just stuff to avoid
*.zip
Expand Down
73 changes: 73 additions & 0 deletions app_inspect_online.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/bin/bash
# Really?!?! Why did I have to write this myself? c'mon Splunk!!
USERNAME=${USERNAME-lalleman}

# jp - jmsepath CLI tool.

if [[ ! -x $(command -v jp) ]]
then
echo "Install the 'jp' command line too. https://github.com/jmespath/jp"
exit 1
fi

set -e

t=$(mktemp)

if [[ -z $token ]]
then
echo "No token found. Logging in."
curl -s -X GET \
-u "$USERNAME" \
--url "https://api.splunk.com/2.0/rest/login/splunk" -o "$t"
cat "$t"
echo
token=$(jp -f "$t" -u data.token)
echo "To reuse the token later, run this in your shell:"
echo "export token=$token"
fi

echo "Uploading $(<.latest_release) to AppInspect via APIs"
curl -s -X POST \
-H "Authorization: bearer $token" \
-H "Cache-Control: no-cache" \
-F "app_package=@\"$(<.latest_release)\"" \
--url "https://appinspect.splunk.com/v1/app/validate" -o "$t"
cat "$t"
echo
request_id=$(jp -f "$t" -u request_id)
echo "Request id: $request_id"

while true
do
curl -s -X GET \
-H "Authorization: bearer $token" \
-H "Cache-Control: no-cache" \
--url "https://appinspect.splunk.com/v1/app/validate/status/${request_id}" -o "$t"

status=$(jp -f "$t" -u status)

echo "STATUS: $status"
#cat "$t"

if [[ $status = "SUCCESS" ]]; then break; fi

sleep 10

done

RPT="app_inspect.report-${request_id}.json"
echo "Downloading report."
curl -s -X GET -H "Authorization: bearer $token" \
-H "Cache-Control: no-cache" \
--url "https://appinspect.splunk.com/v1/app/report/${request_id}" \
-o "${RPT}"

echo "Summary"
jp -f "$RPT" 'reports[0].{name:app_name,description:app_description,author:app_author,version:app_version,hash:app_hash,appinspect_ver:run_parameters.appinspect_version}'

jp -f "$RPT" 'reports[0].summary'

jp -f "$RPT" "reports[0].groups[].checks[].{name:name,description:description,result:result} | [?result!='success' && result!='not_applicable'] | [][name, description, result]"

echo "See $RPT"
47 changes: 47 additions & 0 deletions bin/jpath.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import json
import os
import re
import sys

import splunk.Intersplunk as si
Expand Down Expand Up @@ -37,9 +38,55 @@ def _func_parse(self, s):
except:
return s

@functions.signature({'types': ['array']}, {'types':['string']}, {'types':['string']})
def _func_unroll(self, objs, key, value):
""" What to call this"?
unroll
to_hash Called hash in JSON
zipdict ?
kvarray2hash a bit long
xyseries haha
table/untable
make_hash
unzip Maybe
"""
d = dict()
for item in objs:
try:
k = item[key]
v = item[value]
if not isinstance(k, (str, unicode)):
k = str(k)
k = sanitize_fieldname(k)
# XXX: User option: Overwrite, or make mvlist (Possibly just make 2 different functions?)
if k not in d:
d[k] = v
else:
# Opportunistically turn this into a container to hold more than on value.
# Generally harmful to structured data, but plays nice with Splunk's mvfields
if not isinstance(d[k], list):
d[k] = [ d[k] ]
d[k].append(v)
except KeyError:
# If either field is missing, just silently move on
continue
except Exception as e:
# FOR DEBUGGING ONLY
return "ERROR: Couldn't find key={} value={} in {}".format(key, value, item)
return d


jp_options = jmespath.Options(custom_functions=JmesPathSplunkExtraFunctions())


def sanitize_fieldname(field):
# XXX: Add caching, if needed
clean = re.sub(r'[^A-Za-z0-9_.{}\[\]]', "_", field)
# Remove leading/trailing underscores
# It would be nice to preserve explicit underscores but don't wnat to complicate the code for
# a not-yet-existing corner case. Generally it's better to avoid hidden fields.
clean = clean.trim("_")


def flatten(container):
if isinstance(container, (list, tuple)):
Expand Down

0 comments on commit 411a197

Please sign in to comment.