No Data is Showing in Splunk for Azure Monitor Add On

[CharbelNemnom]

Hello Folks,

I followed exactly the following three posts and I set up successfully AzureMonitorAddonForSplunk.
I have installed the dependencies and I allowed the two ports: 5671/5672.

https://www.splunk.com/en_us/blog/cloud/splunking-microsoft-azure-monitor-data-part-1-azure-setup.html
https://www.splunk.com/en_us/blog/cloud/splunking-microsoft-azure-monitor-data-part-2-splunk-setup.html
https://answers.splunk.com/answers/659163/unable-to-get-azure-activity-log-azure-diagnostic.html

Please note that I don't see any event logged in Splunk so far, however, I can see that my Event Hub Namespace is receiving data.

Here are the errors that I can see on Splunk side:

01-13-2020 14:52:31.764 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" ValueError: Invalid connection string
01-13-2020 14:52:31.796 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/azure_event_hub.py" ERRORInvalid connection string
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" Traceback (most recent call last):
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" File "/data/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/modinput_wrapper/base_modinput.py", line 127, in stream_events
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" self.collect_events(ew)
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" File "/data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" input_module.collect_events(self, ew)
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" File "/data/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 74, in collect_events
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" audit_events = azutils.get_items(helper, access_token, url, items=[])
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" File "/data/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/utils.py", line 33, in get_items
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" raise e
01-13-2020 14:52:32.063 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" HTTPError: 401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2020-01-06T14:52:31.778854Z+and+activityDateTime+le+2020-01-13T14:45:31.978925Z
01-13-2020 14:52:32.092 +0000 ERROR ExecProcessor - message from "python /data/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py" ERROR401 Client Error: Unauthorized for url: https://graph.microsoft.com/beta/auditLogs/directoryAudits?$orderby=activityDateTime&$filter=activityDateTime+gt+2020-01-06T14:52:31.778854Z+and+activityDateTime+le+2020-01-13T14:45:31.978925Z
01-13-2020 14:52:37.562 +0000 WARN DateParserVerbose - Failed to parse timestamp in first MAX_TIMESTAMP_LOOKAHEAD (128) characters of event. Defaulting to timestamp of previous event (Mon Jan 13 14:52:36 2020). Context: source=/data/applications/adobe/aem/author/crx-quickstart/logs/error.log|host=vm-azs-prd-rlxcomv6-301|rlxcom_prod_author_aem_error|161722
01-13-2020 14:52:40.814 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64
01-13-2020 14:52:40.814 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback);
01-13-2020 14:52:40.814 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^
01-13-2020 14:52:40.814 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null
01-13-2020 14:52:40.814 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5)
01-13-2020 14:52:40.815 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5)

Your help is highly appreciated.

Many Thanks!
-Charbel

[CharbelNemnom]

Here are additional error logs specific for: TA-Azure_Monitor

I've noticed that Azure Monitor Metrics (Stream events from Azure resources via Azure Monitor REST API) is working but NOT for: Azure Monitor Diagnostic Logs and Azure Monitor Activity Log.

Your help is highly appreciated.

01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ontimeout (timers.js:386:11)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at tryOnTimeout (timers.js:250:5)
01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timer.listOnTimeout (timers.js:214:5)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback);
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5)
01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" stream.write(buffer, callback);
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ^
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'write' of null
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ontimeout (timers.js:386:11)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at tryOnTimeout (timers.js:250:5)
01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timer.listOnTimeout (timers.js:214:5)

[sebastus]

This error "cannot read property write of null" means that your hubs.json file is incorrect. Rules: 1. hubs.json should list no hubs that don't actually exist 2. it must list any hub that you want to ingest messages from note that I said "hub", not "hub namespace". hubs are listed within the hub namespace.

…

On Wed, Jan 15, 2020 at 7:21 AM Charbel Nemnom ***@***.***> wrote: Here are additional error logs specific for: *TA-Azure_Monitor* I've noticed that *Azure Monitor Metric*s (Stream events from Azure resources via Azure Monitor REST API) is working but NOT for: *Azure Monitor Diagnostic Logs* and *Azure Monitor Activity Log*. Your help is highly appreciated. 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ontimeout (timers.js:386:11) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at tryOnTimeout (timers.js:250:5) 01-14-2020 16:14:13.163 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timer.listOnTimeout (timers.js:214:5) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" /data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" stream.write(buffer, callback); 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" ^ 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" TypeError: Cannot read property 'write' of null 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at ontimeout (timers.js:386:11) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at tryOnTimeout (timers.js:250:5) 01-14-2020 16:14:13.183 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_diagnostic_logs.sh" at Timer.listOnTimeout (timers.js:214:5) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" /data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" stream.write(buffer, callback); 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" ^ 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" TypeError: Cannot read property 'write' of null 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Object.frames.writeFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/frames.js:64:9) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Connection.sendFrame (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/connection.js:329:10) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ReceiverLink.Link.attach (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:152:27) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timeout._onTimeout (/data/splunk/etc/apps/TA-Azure_Monitor/bin/app/node_modules/amqp10/lib/link.js:270:12) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at ontimeout (timers.js:386:11) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at tryOnTimeout (timers.js:250:5) 01-14-2020 16:14:13.322 +0000 ERROR ExecProcessor - message from "/data/splunk/etc/apps/TA-Azure_Monitor/bin/azure_activity_log.sh" at Timer.listOnTimeout (timers.js:214:5) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLHDZVRJ3U2K57GBWDLQ522QPA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEI7KF2I#issuecomment-574530281>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLHAICKSY74OWHCW6ZLQ522QPANCNFSM4KGQWCWA> .

[CharbelNemnom]

Thank you for your help!

I used the default Add-On settings, how can I see the Hub.json file?

On Azure size, the hub(s) under Event Hubs Entities are:

insights-activity-logs
insights-logs-auditlogs
insights-logs-signinlogs

However, the Event Hub Namespace has a different name.

I am ingesting events from 7 Azure subscriptions, and each subscription has it's own Event Hub Namespace which has different name for every subscription, however, the Hub names below are the same across all Event Hub Namespace.
insights-activity-logs
insights-logs-auditlogs
insights-logs-signinlogs

Please advise?

Thank You!
-Charbel

[CharbelNemnom]

Hello, I found the following wiki about updating the hubs.json file. But it's not clear what I want to update in it.

https://github.com/Microsoft/AzureMonitorAddonForSplunk/wiki/Configuration-of-Splunk#hubsjson

Here is my existing hubs.json file that I found in TA-Azure_Monitor/bin/app.

{
"docs01": "DELETE THESE LINES",
"docs02": "",
"docs03": "Edit this file according to details on this page:",
"docs04": "https://github.com/Microsoft/AzureMonitorAddonForSplunk/wiki/Configuration-of-Splunk#hubsjson",
"docs05": "/DELETE THESE LINES"
}

Those are my hubs in the hub namespace.

insights-activity-logs
insights-logs-auditlogs
insights-logs-signinlogs

Could you please advise with an example?

Many Thanks!
-Charbel

[sebastus]

I added a sample to the wiki that you linked above.

…

On Fri, Jan 17, 2020 at 11:17 AM Charbel Nemnom ***@***.***> wrote: Hello, I found the following wiki about updating the *hubs.json* file. But it's not clear what I want to update in it. https://github.com/Microsoft/AzureMonitorAddonForSplunk/wiki/Configuration-of-Splunk#hubsjson Here is my existing hubs.json file that I found in TA-Azure_Monitor/bin/app. { "docs01": "DELETE THESE LINES", "docs02": "", "docs03": "Edit this file according to details on this page:", "docs04": " https://github.com/Microsoft/AzureMonitorAddonForSplunk/wiki/Configuration-of-Splunk#hubsjson ", "docs05": "/DELETE THESE LINES" } Those are my hubs in the hub namespace. *insights-activity-logs insights-logs-auditlogs insights-logs-signinlogs* Could you please advise with an example? Many Thanks! -Charbel — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLCAGBQU2RWDJMPIT7TQ6GHS5A5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHL2KA#issuecomment-575585576>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLDEHY3RJLH4EQAGDI3Q6GHS5ANCNFSM4KGQWCWA> .

[sebastus]

re: "7 azure subscriptions" the TA is designed to store the settings for a single instance. If you add multiple instances of the TA into Splunk, the settings of each new one will over-write the settings previously added. I'm sorry to say I don't have time to add a "multi-instance" capability - perhaps someone from the community will do it. The solutions to monitoring multiple subscriptions: 1. consolidate the output of all of the subscriptions into the event hub namespace of a single subscription 2. use a distinct heavy forwarder for each additional hub namespace

…

On Wed, Jan 15, 2020 at 9:33 PM Charbel Nemnom ***@***.***> wrote: Thank you for your help! I used the default Add-On settings, how can I see the *Hub.json* file? On Azure size, the hub(s) under Event Hubs *Entities* are: *insights-activity-logs insights-logs-auditlogs insights-logs-signinlogs* However, the Event Hub Namespace has a different name. I am ingesting events from 7 Azure subscriptions, and each subscription has it's own Event Hub Namespace which has different name for every subscription, however, the Hub names below are the same across all Event Hub Namespace. *insights-activity-logs insights-logs-auditlogs insights-logs-signinlogs* Please advise? Thank You! -Charbel — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLEV45WPZCZ6KYNIG33Q556JZA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJB4GII#issuecomment-574866209>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLDSXRC7MCTGVPOHSULQ556JZANCNFSM4KGQWCWA> .

[CharbelNemnom]

Thank You!

How to get the resource ID for each one please?

insights-activity-logs
insights-logs-auditlogs
insights-logs-signinlogs

[sebastus]

That is Hard-coded, just as in the example.

…

On Fri, Jan 17, 2020, 6:51 AM Charbel Nemnom ***@***.***> wrote: Thank You! How to get the *resource ID* for each one please? *insights-activity-logs insights-logs-auditlogs insights-logs-signinlogs* — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLGRCOJTXHL6VGVWY43Q6GSTVA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHST5A#issuecomment-575613428>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLAAGD3UBXCTWFORHTLQ6GSTVANCNFSM4KGQWCWA> .

[CharbelNemnom]

Thank you so much for your fast response!

Here is my hubs.json file look like. Is it correct?

{
"insights-activity-logs": "resourceId",
"insights-logs-auditlogs": "resourceId",
"insights-logs-signinlogs": "resourceId"
}

In regards to the following point, do you have more details on how to do it?
At the moment, I have an event hub namespace per subscription (7).

1. consolidate the output of all of the subscriptions into the event hub namespace of a single subscription.

[sebastus]

Hubs.json looks fine. To consolidate, in Azure when you specify the namespace and hub, choose one from a single subscription. Might not be possible in all cases. I don't remember all of the details. Q for you: where did "insights-activity-logs" come from? Usually, the activity logs go into a platform-named hub called "insights-operational-logs". Maybe that changed in the platform recently.

…

On Fri, Jan 17, 2020, 7:19 AM Charbel Nemnom ***@***.***> wrote: Thank you so much for your fast response! Here is my *hubs.json* file look like. Is it correct? { "insights-activity-logs": "resourceId", "insights-logs-auditlogs": "resourceId", "insights-logs-signinlogs": "resourceId" } In regards to the following point, do you have more details on how to do it? At the moment, I have an event hub namespace per subscription (7). 1. consolidate the output of all of the subscriptions into the event hub namespace of a single subscription. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLFA5PM7ATBDFNIIOPTQ6GV4NA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJHUZUY#issuecomment-575622355>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLHLWOO4HP5J65MCILLQ6GV4NANCNFSM4KGQWCWA> .

[CharbelNemnom]

Thank You!

Yes, this was changed in the platform recently with the new activity log.
https://docs.microsoft.com/en-us/azure/azure-monitor/platform/diagnostic-settings-legacy

I have the "insights-activity-logs" in each of my Event Hub for every subscription. I enabled Diagnostics Settings for each azure subscription and selected the corresponding Event Hub.

[CharbelNemnom]

Hello,

Could you please confirm the design below and confirm if it works for monitoring multiple subscriptions based on option 1 mentioned above. Thank You!

1. Consolidate the output of all of the subscriptions into the event hub namespace of a single subscription

[sebastus]

That's the general idea, but it looks like you plan to send data to event hubs in the subscriptions, then from those event hubs to another consolidator. My thought was that you would (in the diagnostic profile of each resource) designate the destination hub as the consolidator, without the middle step. The function can send directly to your splunk cloud instance. You might have to ask Splunk support to switch on HEC.

…

On Tue, Jan 21, 2020 at 3:43 PM Charbel Nemnom ***@***.***> wrote: Hello, Could you please confirm the design below and confirm if it works for monitoring multiple subscriptions based on option 1 mentioned above. Thank You! 1. *Consolidate the output of all of the subscriptions into the event hub namespace of a single subscription* [image: image] <https://user-images.githubusercontent.com/13448198/72819122-f56e4300-3c6c-11ea-8d2c-be6b4dc387a6.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLBLN2U5NGRVIYWNGT3Q64J2XA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQGDKA#issuecomment-576741800>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLBTWAQKB6XQHX3IBGLQ64J2XANCNFSM4KGQWCWA> .

[CharbelNemnom]

Thank you Greg,

I misunderstood your point about point 1: Consolidate the output of all of the subscriptions into the event hub namespace of a single subscription.

I thought that I want to send the data to event hubs in every subscription, then from those event hubs to another consolidator instead of connecting 7 times to Splunk, since I have 7 Azure subscriptions that I want to monitor.
If I understood your point here, is to have ONLY one event hub namespace for all 7 subscriptions, and then in the diagnostic profile of each resource, assign the destination hub as the consolidator.

Now I have 7 Data inputs under Azure Monitor Diagnostic Logs in Splunk.

Could you please elaborate more?

Many Thanks Greg!
-Charbel

[sebastus]

There should only be one data input for the consolidated feed from all 7 subscriptions. You'll need to filter based on subscription id to separate the feeds in your queries.

…

On Tue, Jan 21, 2020 at 4:27 PM Charbel Nemnom ***@***.***> wrote: Thank you Greg, I misunderstood your point about point 1: Consolidate the output of all of the subscriptions into the event hub namespace of a single subscription. I thought that I want to send the data to event hubs in every subscription, then from those event hubs to another consolidator instead of connecting 7 times to Splunk, since I have 7 Azure subscriptions that I want to monitor. If I understood your point here, is to have ONLY one event hub namespace for all 7 subscriptions, and then in the diagnostic profile of each resource, assign the destination hub as the consolidator. Now I have *7 Data inputs under Azure Monitor Diagnostic Logs in Splunk*. Could you please elaborate more? Many Thanks Greg! -Charbel — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#172?email_source=notifications&email_token=AAJ2XLDTESMA3FIUKVRNQ43Q64O7ZA5CNFSM4KGQWCWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJQLITI#issuecomment-576762957>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJ2XLAAZB36PW4HVJTDWGTQ64O7ZANCNFSM4KGQWCWA> .