Add support for OAuth

[jorritfolmer]

Because e.g. Gmail is dropping support for user/pass authentication:
https://gsuiteupdates.googleblog.com/2019/12/less-secure-apps-oauth-google-username-password-incorrect.html

[rickmyers-txstate]

+1
O365 is moving in this same direction.

[hkelley]

Microsoft has set a date of October 2022.

Today, we are announcing that, effective October 1, 2022, we will begin to permanently disable Basic Auth in all tenants, regardless of usage

https://techcommunity.microsoft.com/t5/exchange-team-blog/basic-authentication-and-exchange-online-september-2021-update/ba-p/2772210

Assuming the add-in is only collecting reports on a daily basis, the code can probably be kept simple and throw away the access token after each fetch (it will have expired by the time we reconnect to the mailbox). We just need to securely persist the access token and the initial secret (or just get a new access token each day)?

https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth

[jorritfolmer]

Thanks for the heads up! Getting a token from the Azure AD authorize endpoint should be easy using the MSAL Python lib. I just verified the IMAPClient library from Menno Finlay-Smits supports the XOAUTH2 method needed to submit tokens against the IMAP service. Looks like a fun little project for the summer to get it working for O365 and Google Workplace.

[hkelley]

This Microsoft deadline seems to be holding for October so I expect the O365 mailbox pickups to break soon. Have most other people switched to file-based pickup?

[hkelley]

If I were to take a crack at adding this capability:

a) Is there an AOB-friendly version of the TA could import to my local Splunk dev environment?

b) Is there an approach you had planned? Given the extra complexity/config options required for OAuth, I was envisioning a new/separate input type: dmarc_imap_oauth. That would allow for extra fields to collect those options, e.g.:

    "authority": "https://login.microsoftonline.com/Enter_the_Tenant_Name_Here",
    "scope": ["https://graph.microsoft.com/.default"],
    "endpoint": "https://graph.microsoft.com/v1.0/users"

Then a small refactoring of the Imap2Dir.get_dmarc_messages would allow basic or OAuth (based on the presence of the settings above)

[jorritfolmer]

Awesome, sounds good!
I've updated the repo to include TA-dmarc.aob_meta which should allow you to import into AoB using these steps:

1. Create a .tar.gz file from the TA-dmarc/ directory so a tar -ztvf shows the path starting at the TA-dmarc/ level.
2. Import in Splunk AoB.

[hkelley]

Package imported. I had to fiddle with the tar commands a bit to exclude .git files. For my own future reference, the (Windows) commands are:

SET ARCHIVE=%CD%\ta-dmarc-add-on-for-splunk_AOB.tar.gz

pushd \<PARENT_FOLDER_OF_TA>

DEL %ARCHIVE%
tar -cz   --exclude .git --exclude .gitignore -f %ARCHIVE%  TA-dmarc

popd

REM open to verify that the app folder is at the top of the archive
tar ztvf %ARCHIVE% | more

Any thoughts on the code structure/approach from b) above?

[jorritfolmer]

b) Is there an approach you had planned? Given the extra complexity/config options required for OAuth, I was envisioning a new/separate input type: dmarc_imap_oauth. That would allow for extra fields to collect those options, e.g.:

    "authority": "https://login.microsoftonline.com/Enter_the_Tenant_Name_Here",
    "scope": ["https://graph.microsoft.com/.default"],
    "endpoint": "https://graph.microsoft.com/v1.0/users"

Then a small refactoring of the Imap2Dir.get_dmarc_messages would allow basic or OAuth (based on the presence of the settings above)

Yes, agreed on all counts!

[hkelley]

Unrelated to the OAuth matter, I'm trying to learn how to prepare add-ons for GitHub. Which version of the aob_meta file is the one you added to this project?

When I look in a different add-on I'm building, I see three <ADDON_NAME>.aob_meta files in:

 /etc/apps/splunk_app_addon-builder/local/validation/TA-NAME
 /var/data/tabuilder/package/TA-NAME
 /var/data/tabuilder/export_ta/TA-NAME

The first two seem to be updated when I run the AOB UI's Validate operation. The second two are updated when I run the Export operation. Based on this, I'm assuming that /var/data/tabuilder/package/TA-NAME (always updated) is the one to grab for inclusion in a share-able project.

[jorritfolmer]

The addon builder generates everything in the TA-dmarc app you need to run it anywhere. This doesnt include the aob meta stuff but thats only needed to modify the addon forms. I would suggest to forget about the aob meta stuff until you have a working version, and only then (once) grab the aob meta file from the “export add-on” button in the builder.

The only thing to be aware of it to regularly move conf files and other stuff from the local/ directory to the default/ before comitting. Local/ is included in .gitignore. To verify if I have everything I usually do a git clone on a separate empty and fresh Splunk instance to see if I’ve included everthing in default/.

[hkelley]

You are right. I found the AOB maddening and never could get on-disk .conf changes to to appear in the UI. I was trying to copy the existing dmarc_imap input to dmarc_imap_oauth2. I gave up and started work on the OAuth2 bits.

First draft here:

https://github.com/jorritfolmer/TA-dmarc/compare/master...hkelley:imap-oauth2?expand=1

Some items to note:

• I am assuming that the client-authenticated flow (which is most similar to a username+password for a service account) will be used
• For Office 365 (and perhaps others), this requires specifying the mailbox, which is no longer the same as the username. Therefore, I repurposed the existing opt_imap_mailbox (e.g. dmarc-reports@xyz.com) and added the opt_imap_folder (e.g. INBOX).
• I kept all IMAP changes in imap2dir.py. I have imported msal but not included it in my PR.
• I probably spent more time figuring out the O365 configuration options than I did the coding. I can try to merge some of this into the readme or another spot you suggest

[jorritfolmer]

Hah! Looks good! Some short pointers in the readme would be nice indeed, as well as including the msal library under bin/. I’ll do some testing this weekend on my Google Workplace account.

[hkelley]

I've updated the readme and added msal. I also took a crack at "re-wiring" the UI parameters for the existing IMAP basic input so that they will work with the new imap2dir.py methods.

If all works as hoped, the existing inputs will behave as-is and you can simply:

1. clone the dmarc_imap input as dmarc_imap_oauth2
2. add the params for mailbox, authority, and scope to the cloned input

[jorritfolmer]

Thanks! I've added the new input based on your commits in the oauth2 branch.

My time ran out so I only managed to test whether or not the old IMAP input works in a fresh install and upgrade scenario and it does. However, I had to revert your imap_folder rename to prevent breaking old IMAP inputs in upgrade scenario's.
That leaves the new imap_mailbox field which I ended up renaming to imap_username as it most closely resembles a username to access the mailbox with the previously acquired token.

Can you checkout and test the oauth2 branch with your O365 setup?

[hkelley]

I'm sure this is a rookie mistake, but I can't seem to get the TA installed. I have:

1. Uninstalled+reinstalled Splunk
2. Cloned the branch you created

# git status
On branch oauth2
Your branch is up to date with 'origin/oauth2'.

3. Tar'd it into a file for import

tar -czf \temp\TA-dmarc.tar.gz TA-dmarc

5. Imported the .gz file. Receive this Message

Unable to initialize modular input "dmarc_pop3" defined in the app "TA-dmarc": Introspecting scheme=dmarc_pop3: script running failed (exited with code 255)..

The app partially installs but the Add button for a credential is missing.

On the bright side, my outside-of-Splunk test harness runs fine with the imap2dir.py package.

[jorritfolmer]

You can skip the tar step and git clone directly into etc\apps.

But TBH it looks like an issue with the TA. Not sure what exactly, it worked yesterday on Linux and a fresh Splunk 8.2.5.

[hkelley]

I will try it on v8. This testing was on v9 (Windows)

[hkelley]

Trying to keep it simple, I grabbed Splunk v8 and the latest TA-dmarc from Splunkbase. Installing the app on Windows, I am able to add a credential but not to create an input. I don't get any visible error messages.

Any idea which logs I should be peering into?

After I upgrade/overwrite to the new app version, the same error from Splunk v9 appears:

Unable to initialize modular input "dmarc_pop3" defined in the app "TA-dmarc": Introspecting scheme=dmarc_pop3: script running failed (exited with code 255)..

[hkelley]

I tried swapping the sequence of the inputs in inputs.conf.spec. Whichever one is first in the file (after dmarc_directory, which is fine) is the one reported in the UI's messages when I restart Splunk and open the app.

Unable to initialize modular input "dmarc_imap_oauth2" defined in the app "TA-dmarc": Introspecting scheme=dmarc_imap_oauth2: script running failed (exited with code 255)..

[jorritfolmer]

I can't reproduce the errors on a fresh Windows 2016 with fresh Splunk 8.2.8.

1. Next -> Next install Windows 2016
2. Next -> Next install Git (to get gitbash)
3. Next -> Next install Splunk 8.2.8
4. git clone https://github.com/jorritfolmer/TA-dmarc.git in c:\program files\splunk\etc\apps
5. git checkout oauth2
6. Restart Splunk
7. Configured IMAP input with basic auth
8. Verified with search index=_internal sourcetype="ta:dmarc:log"

[hkelley]

Glad to hear that. I'll get a clean VM and retry (assuming we still need to verify the OAuth import).

[jorritfolmer]

Please do, I tried to use OAuth2 against my Google Workplace account, but the msal library won't allow it saying:

ValueError: Unable to get authority configuration for https://accounts.google.com/o/oauth2/v2/auth. Authority would typically be in a format of https://login.microsoftonline.com/your_tenant Also please double check your tenant name or GUID is correct.

This was after adding validate_authority = False to the msal.ConfidentialClientApplication() statement. So somewhere in the future there will be a need to replace the msal library with a generic oauth2 library, or maybe vendor specific IMAP Oauth2 inputs.

I'll wait for your confirmation that the current setup works against O365 before publishing to Splunkbase.

[hkelley]

I have used a brand new Windows server and a fresh install of Splunk. I'm still having the same issue.

If I install the app from the tar file, I get a timeout while installing.

Error connecting to /services/apps/local: The read operation timed out

If I manually copy the app, I get this Message in the UI.

Unable to initialize modular input "dmarc_imap_oauth2" defined in the app "TA-dmarc": Introspecting scheme=dmarc_imap_oauth2: script running failed (exited with code 255)..

I tried swapping the sequence of the inputs in inputs.conf.spec. Whichever one is first in the file (after dmarc_directory, which is fine) is the one reported in the UI's Messages when I restart Splunk and open the app.

[jorritfolmer]

Can you try to use the source from Github? That way we can eliminate a number of variables. My suggestion would be to install Git from https://git-scm.com/download/win so you can omit a number of tar and/or copy steps.

• delete the entire TA-dmarc directory
• git clone https://github.com/jorritfolmer/TA-dmarc.git in c:\program files\splunk\etc\apps
• git checkout oauth2
• restart Splunk

[hkelley]

The direct git checkout worked. I'm not sure why a file copy from my workstation's folder (on the same branch) didn't work, but I'll be the first to admit that I'm still learning a lot about Splunk dev practices.

I have it working now (still need to figure out why they aren't being indexed, but that's probably a config error of mine)

tid=MainThread file=base_modinput.py:log_info:295 | get_dmarc_messages: 76972 messages in folder Inbox

Two fixes are needed (this may also address the Google login failures you saw). I also have a lot of extra debug logging in imap2dir.py that we might as well include. I checked it into my fork.

Fixes:

• input_module_dmarc_imap_oauth2:69 Change to
  opt_oauth2_scope = helper.get_arg('oauth2_scope')
• Fix description under authority, Office 365 does not want the /v2.0 suffix

[hkelley]

Data ingestion issue found and logged at #43 . Unrelated to OAuth2 changes, just a side-effect of my testing on Windows.

[jorritfolmer]

Thanks! I've seen you committed the helper.get_arg fix. Can you be more specific what the authority url example should be in the setup screen?

It looks that's the final issue to clear before releasing this to Splunkbase?

[hkelley]

In the case of Office 365, the desired authority and scope are:

opt_oauth2_authority      https://login.microsoftonline.com/<tenant_id>
opt_oauth2_scope          https://outlook.office365.com/.default

I may have previously mentioned a v2 for the authority, but in my testing, that is not accepted by O365

[jorritfolmer]

Thanks again for your efforts! I've updated the screens and docs. v4.1.0 is pending validation and cloud vetting on Splunkbase. It should be available somewhere next week for download. In the meantime, it's available here on Github in master with release tag 4.1.0.

[jorritfolmer]

Update: the new release won't make it to Splunk Cloud because Splunkbase found some library vulnerabilities in the stuff that the Splunk add-on builder generates. See #45. This doesn't seem to be easily fixable without a new add-on builder release.

[rickmyers-txstate]

Notes. For office365 the username and password should be replaced with Application ID and secret and not username/password.

I found that imap2dir.py line 131 calls msal.ConfidentialClientApplication which indicates "Acquires token for the current confidential client, not for an end user."

I am also not getting data into splunk after switching over to oauth.
I have enabled debug logging and ta_dmarc_dmarc_imap_oauth2.log looks like it is processing data.

[jorritfolmer]

Yes the Splunk add-on builder has fantastic feature for entering, storing and using credentials. You can add an account tab to an add-on through a single click! However that's all the customisability there is, so we'll probably have to live with the fact that for client id/secret the terminology doesn't match. To compensate, it is covered in the README that user/pass should be read as client id/secret.

Can you share more details about the IMAP Oauth2 input not working properly?

[rickmyers-txstate]

It was not clear looking at README to use an application id and secret.

As far as not working. I have not seen any DMARC reports injected into splunk since I switched from basic auth to OAuth. I have enabled debugging within the application.

oh wait!. I am wondering if it is starting back from the beginning of the mailbox......... and not limiting to things ( say unread )

looking at the logs ta_dmarc_dmarc_imap_oauth2.log it does look like it is processing message from back in 2019. Well. looks like I will have a while to wait. or I can move things to a new folder. I will report back sorry.

[jorritfolmer]

So to summarize the current state of this issue:

• OAUTH2 works for O365
• OAUTH2 doesn't work for Google Workspace. However, Google have suspended the deprecation of Less Secure Authentication, for now (source: https://workspaceupdates.googleblog.com/2020/03/less-secure-app-turn-off-suspended.html)
• TA-dmarc v4.1.1 available here and on Splunkbase for on-prem and cloud

I'll leave this issue open to track future work for Google Workspace or others.

[hkelley]

Thank you. Mine is deployed to Splunk cloud and is ingesting now. Am I correct that the KV Store "history log" is global, so my new input (using OAuth2) won't duplicate any reports that the old input had already ingested?

In case it helps anyone else making the transition from BASIC to OAuth, an error message like the following (visible from index=_internal sourcetype"ta:dmarc:log"), might mean that you have an old O365 client access rule blocking your auth to the mailbox (even though Azure AD authentication of the service principal is working).

imaplib.IMAP4.error: SELECT command error: BAD [b'User is authenticated but not connected.']

[jorritfolmer]

Am I correct that the KV Store "history log" is global, so my new input (using OAuth2) won't duplicate any reports that the old input had already ingested?

I think the reports will be duplicated. It will only skip already ingested messages if the servername and username haven’t changed. The key on line 319 takes server, user and uid:

TA-dmarc/bin/dmarc/imap2dir.py

Lines 318 to 323 in ba207ca

	for uid in messages:
	key = "%s_%s_%d" % (self.opt_imap_server,
	self.opt_global_account["username"], uid)
	if self.helper.get_check_point(key) is not None:
	seen_uids.add(uid)
	new_uids = set(messages) - seen_uids

Unfortunately the key will be different since you need to specify a client id for Oauth2 and an email for plain IMAP.