OSStatus error -34018 when accessing keychain during sending e-mail phase

[everetteallen]

10/27/14 10:18:09.300 AM AutoPkgr[34833]: SecOSStatusWith error:[-34018] The operation couldn’t be completed. (OSStatus error -34018 - Remote error : The operation couldn‚Äôt be completed. (OSStatus error -34018 - client has neither application-identifier nor keychain-access-groups entitlements))

[eahrold]

@everetteallen Thanks for the report.
I think this is an issue with AutoPkg not being sandboxed (and maybe Mavericks specific). I believe this is somewhat common across apps that access the security framework. It shouldn't effect the functioning of AutoPkgr.

Are you having any specific run time problems?
Can you tell if this is raised with any particular action?

Here's a similar discussion about the same error code...
https://gist.github.com/hjuutilainen/7544223

[MScottBlake]

I am seeing the same error in 10.9.5 and 10.10 when I try (and fail) to connect it to a JSS.

[eahrold]

@MitchelSBlake thanks for letting us know.
I'm bummed this didn't get worked out in 10.10.

Actually I think it happens when accessing the keychain to retreive the password to send the e-mail indicating that an error occurred. It's the kc accessing the security framework that is triggering the error.

If we're able to logistically figure out how to add a provisioning profile and at the same time keep this open source, we should be able to set up the keychain-entitilements necessary to silence this warning.

That being said, if you wouldn't mind, try running a similar autopkg invocation and see if you get the same error there.

Thanks

[eahrold]

Hey Scott ( @MitchelSBlake ),
I just put a branch up called Keychain-Entitilements

I have no Idea how it will work for open source contributors, would you mind cloning and trying to build off that branch and see if it causes any problems.

Thanks.

--Eldon

[MScottBlake]

It fails when I build on the mailcore2 step.

WARNING: could not get prebuilt.list from repository
cat: prebuilt.list: No such file or directory
building file list ... rsync: link_stat "/Users/msblake/Downloads/autopkgr-Keychain-Entitilements/mailcore2/build-mac/../Externals/ctemplate/lib/." failed: No such file or directory (2)
done

[MScottBlake]

Turns out, this isn't my problem. I'm still seeing the errors, but they aren't causing a failure in the software.

My problem turns out to be that JDS servers are not reported through the API, so Autopkgr cannot find them.

[eahrold]

Thanks Scott,
I've got another branch updating the mailcore2 frameworks. So we may be two (or more) steps away from working out the initial issue here. Cursed Entitlements!

As far as your JDS, what is not getting reported? Is the DistPoints or the server itself?

[MScottBlake]

In my JSS API, when you do a GET for distributionpoints, JDS servers are not listed. I inherited an environment that is 100% JDS, so when I enter my user/pass into Autopkgr, "nothing" happens. However, when I look at the tomcat logs, the server is returning http code 200, it's just an empty list.

It looks as if Autopkgr will need to either come up with a different way of pulling JDS servers, or wait until JAMF updates the API to list them, but that should probably be broken out into a new issue.

[eahrold]

Hey Scott,
With your API how would you normally get a list of distribution points? I can look into fixing that (or if you want to put up a PR that'd be awesome!)

I'm not sure if the jss-autopkg-addon actually supports JDS yet, but I think the python-jss lib is about to get an overhaul on how distribution points are handled which may address that.

On another note,
I did just realize that for some reason mailcore2 actually has the prebuilt.list file in it's .gitignore
I just pushed a change to override that which may allow you to build.
Keep me posted.

[MScottBlake]

I would get distribution points the same way as anyone else, but JDS servers are not yet exposed to the API.

[MScottBlake]

Still cannot build. Current error:
sed: 2/mailcore2/scripts/../Externals/builds/workdir/20141102225057/src/icu/source/common/unicode/uvernum.h: No such file or directory

[eahrold]

That one's strange. I'm looking forward to mailcore2 getting their os x podspec working.

Look in the mailcore2/scripts folder there should a file called get-mac.sh, just run that from the terminal it should give you a more verbose output than when it's run from Xcode.

[MScottBlake]

The only get-*.sh file is get-prebuilt.sh. Running that, I get this:
WARNING: could not get prebuilt.list from repository

I might suggest clearing out your repo, re-cloning, then adding in files until you can build.

[eahrold]

Hey Scott,
Sorry, I was working on another branch and thought I had updated the Mailcore2 framework on this one. I just did update it, so you should be able to pull and get things running.

You may need to fiddle with the code signing identity in the build settings to make it work.
Thanks for helping work through this.

[MScottBlake]

Sorry, I'm not going to be able to build for at least for a few weeks because I don't currently have a code signing certificate. I already requested a developer account through my employer weeks ago, but the request still needs to go through some red tape.

If you have a specific build you think will fix the issue, I'd be happy to test the app if you can get it to me.

[eahrold]

No problem. I'll get a build put up soon.

The other work around if you're interested is to just create a self signed certificate in
Keychain Access -> Certificate Assistant -> Create a Certificate

Set the name to "Mac Developer" and set the Certificate Type to Code Signing

When you do finally get your certificate make sure you delete the one you create otherwise it can break the automatic id setting when it finds multiple matching.

[MScottBlake]

If only it were that easy. Xcode needs an Apple ID that's a member of the program.

[eahrold]

That's what I was afraid of. I've been able to do that when the app is just code signed, but the provisioning profile may be too much. (However I did realize that it won't work with "Mac Developer" as the name but will with "Local Mac Developer" as the name.)

On another note looks like you JDS issue may be getting addressed in the python-jss lib at least.
https://github.com/sheagcraig/python-jss/releases

Now we will have to figure out how to get that info into AutoPkgr.

[everetteallen]

I am also having the same issue trying to build any branch. The error throws for me on 10.9 and 10.10 as well. Is there are nightly or test binary ?

[eahrold]

@everetteallen which error? The mailcore2 or the Developer ID one?

[eahrold]

@MitchelSBlake
@everetteallen

I just put up another couple commits to the keychain-entitlements in an attempt to make this work.

here is a link to an extremely rough sketch of the contrib docs
https://github.com/lindegroup/autopkgr/blob/Keychain-Entitilements/Docs/Contributing.md

It won't address the SecOSStatusWith issue directly, since that will only get solved when we do release builds with both a code signed binary and keychain entitlements that are signed with the official provisioning profile.

I don't currently have the provisioning profile needed to build testing candidates, but am working with the owners of the project to procure it. When that happens I will get something up for testing.

Mainly what I am trying to do here is make it so the general public can clone, build and contribute.
This most recent build is set to only include the entitlements file when building as release.

I would greatly appreciate it if you would clone, and try to build the keychain-entitlements branch and let me know if you encounter any specific issues. If you do encounter any issues, but can figure out how to resolve them please put up a PR.

Thanks for all of your help,

Eldon

[everetteallen]

Eldon,
I will try the clone when I get a sec. I did have a different experience today. I started with a clean 10.10 os that had never had autopkg, autopkgr, JSS-AutoPKG-Plugin, or Python-JSS installed. For fun I pulled down AutoPKGR and let it do all the installs of git, autopkg and JSSAutoPKG-Plugin/Python-JSS and everything worked as it should except AutoPKGR gave failed error on .jss recipes due to not finding JSS_URL, API_USERNAME, and API_PASSWORD in com.github.autopkg. I when I set these with defaults everything works. I would have thought that these would have been set when filed into the Casper Suit Integration section?
Either way I am not see the error. I did verify that I can reproduce the sequence by deleting
/Applications/AutoPKGR
/Library/Python/2.7/site-packages/jss
/Library/AutoPkg/autopkglib/JSSImporter.py
~/Library/AutoPkg
and then letting a new copy of AutoPKGR do all the installs again and writing the 3 defaults.
So it is reproducible on "dirty" systems.

[eahrold]

I think we have a design issue right now which we'll have to rethink.

Right now the defaults aren't actually set until you've successfully connected to the JSS Repo and retreived an array of distribution points from the api.

However with a JDS setup, no DP's are returned from the api, it looks to AutoPkgr like it's not actually a valid JSS server (even though it is).

We should probably just set the defaults when entered even thought the values may not be correct.

We're working on getting ready for the new python-jss that should have JDS support and once that happens all will be dandy.

I'll put up that branch and post more on that once I have the rough sketch cleaned up.

Thanks again for your help. We really depend on comments like this to help us understand how it's working in a wild.

--Eldon

[eahrold]

@everetteallen I just updated the title because the OSStatus error -34018 is actually a red herring, and unrelated to the JSS. The reason you see it when the JSS fails is because AutoPkgr immediately sends an email about the error. We'd still love it if you had time to take a look at the keychain entitlements branch, see this post on getting set up https://github.com/lindegroup/autopkgr/blob/Keychain-Entitilements/Docs/Contributing.md

[everetteallen]

@eahrold been pulled a way this week on other things but trying to get back to it today. I will see if I can get a build and follow up.

[eahrold]

@MitchelSBlake good news, we've got a pre-release up with JDS support. We'd love it if you could give it a test drive and report back at issue #216.

@everetteallen not sure if you're running JDS as well, if so we would also love your input.

--Thanks

[everetteallen]

So got a chance to do some testing with the pre-release:
Running Shea's firefox.jss recipe on 10.10.1 against a jamfcloud.com hosted system get:
Error running recipes
A Python exception occurred during the execution of autopkg, see the console log for more details. [ERROR] jss.jss.JSSGetError: JSS Error. Response Code: 404 Response"

So I thought this might be related to self-signed cert issue and I did the

got:
Error running recipes
/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/contrib/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.) InsecureRequestWarning) A Python exception occurred during the execution of autopkg, see the console log for more details. [ERROR] jss.jss.JSSGetError: JSS Error. Response Code: 404 Response"

So may be I am confused about the JDS setup. If I have a JSS configured why do I need to add a JDS distribution point? The files, etc should all be uploaded to the JSS anyway, which then pushes them to the JDS. Given this I configured the JSS and JDS info the same. If that is not the case then would explain the 404.

Also still seeing these:
11/26/14 1:01:55.193 PM secd[501]: securityd_xpc_dictionary_handler AutoPkgr[15080] copy_matching The operation couldn’t be completed. (OSStatus error -34018 - client has neither application-identifier nor keychain-access-groups entitlements)
11/26/14 1:01:55.193 PM AutoPkgr[15080]: SecOSStatusWith error:[-34018] The operation couldn’t be completed. (OSStatus error -34018 - Remote error : The operation couldn‚Äôt be completed. (OSStatus error -34018 - client has neither application-identifier nor keychain-access-groups entitlements))

[eahrold]

So first off these messages....

11/26/14 1:01:55.193 PM secd[501]: securityd_xpc_dictionary_handler AutoPkgr[15080] copy_matching The operation couldn’t be completed. (OSStatus error -34018 - client has neither application-identifier nor keychain-access-groups entitlements)
11/26/14 1:01:55.193 PM AutoPkgr[15080]: SecOSStatusWith error:[-34018] The operation couldn’t be completed. (OSStatus error -34018 - Remote error : The operation couldn‚Äôt be completed. (OSStatus error -34018 - client has neither application-identifier nor keychain-access-groups entitlements))

Are completely unrelated to JSS in any way. They're 100% related to accessing the keychain when sending an email. You'll see the exact same thing in the log if you do a "Send Test Email". We're still debating how to best handle including entitlements while keeping the project Open Source. Once we start building the project with entitlements, that error will go away, but that may still be a while.

More to the point.
I may be wrong here, but I don't think the JSS actually uploads files to the JDS. Casper Admin may be responsible for that, but not the JSS REST API.

The reason that currently you need to configure the JDS manually is because it's not exposed to the API, so python-jss has no way of auto-detecting that the way it will with SMB/AFP shares (However Casper Admin uses some private methods to pull this).

The credentials may or may not be the same for your JSS and your JDS, and the URL is almost certainly not. You should be able to find the url for the JDS here https://your.jssserver.com/distributionServer.html?list

@homebysix may be able to chime in, he's got a much more comprehensive knowledge about JSS than I do, but hopefully this will give you a little something more to go on.

Also it may be helpful if you could post the exception traceback from the console.

Thanks for your help.
--Eldon

[everetteallen]

Understood on the email error.

On the JSS 404, there is plenty of mis-understanding about JSS/JDS.
So Shea, Forest and I have actually verified that currently Casper 9.x
does indeed upload the files to the JSS database and then sync down
to the Master JDS. Here is part of an email from Forest's tests (also
noted in the response change with some screen shots for pyton-jss issue no. 1):

"It looks like Casper Admin (both web version and the application) will
just upload the package to tables in the JSS database. There is a table
for the metadata (downloadable_file_chunk_metadata) and another table
for the package data broken into chunks (downloadable_file_chunk_data).
The JDS parent servers just run a cron job every 5 minutes to check for
new packages and download them to their JDS CasperShare in /usr/local/jds."

We were all mis-led at first by the Casper Admin fat app which mounts
the webdav share via https when it runs(like it did when we used
afp/smb shares, likely a JAMF wanted a consistent feel for end users).
In a JDS setup this is for convenience only and even the fat app does
not write to the share.

So to me a JDS should not be treated as a webdav share as we are
treating it now.

I wonder if it makes sense to use the JDS code as code for webdav
(https) servers more generally and then establish the JDS type that uses
the a "db file upload" method as a check box under the JSS
configuration. I believe this is in place in python-jss but suffers the
issue #5 that Shea references at the bottom of
jssimporter/python-jss#1 and is Blocked
according to the latest commit.

Stepping back to a wider view that covers AutoPKGR we need to make sure
that we present representative UI for the JSS/JDS sync in the Casper
Suite integration section that does not confuse the end user.

Also I think the scope of this has changed so might want to open in another issue.
The traceback from the console is attached. I will try to get the name and password for my CasperShareDAV and test with that to confirm working.

Everette

---

11/26/14 1:01:55.189 PM AutoPkgr[15080]: (FULL AUTOPKG TRACEBACK)
File "/usr/local/bin/autopkg", line 1334, in
sys.exit(main(sys.argv))
File "/usr/local/bin/autopkg", line 1328, in main
exit(subcommands[verb]'function')
File "/usr/local/bin/autopkg", line 1152, in run_recipes
autopackager.process(recipe)
File "/Library/AutoPkg/autopkglib/init.py", line 466, in process
self.env = processor.process()
File "/Library/AutoPkg/autopkglib/init.py", line 295, in process
self.main()
File "/Library/AutoPkg/autopkglib/JSSImporter.py", line 634, in main
ssl_verify=sslVerify, repo_prefs=repos)
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 169, in init
self.distribution_points = distribution_points.DistributionPoints(self)
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/distribution_points.py", line 85, in init
self.dp_info = self.jss.DistributionPoint().retrieve_all()
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 317, in DistributionPoint
return self.factory.get_object(DistributionPoint, data)
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 454, in get_object
result = self.jss.get(url)
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 202, in get
self._error_handler(JSSGetError, response)
File "/Library/Python/2.7/site-packages/python_jss-0.4.2-py2.7.egg/jss/jss.py", line 189, in _error_handler
raise exception
jss.jss.JSSGetError: JSS Error. Response Code: 404 Response"

---

[MScottBlake]

Just yesterday, my TAM sent me an alternative workflow for adding packages to a JDS:

1. Create the package in the database using the API.
   • The easiest method is to perform a GET on an existing package, save the XML and modify it. I attached it to the email.
   • Update the XML to include the correct name and display name for the package (case sensitive). You can supply a valid category or set other package preferences in the XML, or the changes can be made from Casper Admin or the web app after posting the XML
   • The post is done with a command similar to the following: curl -k -v -u username:password https://JSSURL.com:8443/JSSResource/packages/name/PackageName -T "/path/to/packages.xml" -X POST
2. Copy the package to /Library/JDS/shares/CasperShare/ (mac location, different on linux)
3. Update the JDS inventory by running 'sudo jamfds inventory'

[everetteallen]

All,
Yes I have seen this "backside injection" method and tried it but I
could never make the JDS Master Distribution Point sync to the other JDS
servers in the chain automatically...yes could then just copy to all and
update inventory but defeats the purpose and design of JDS sync. I have
1 master and 6 subordinate JDS servers at this point and sync works
without fail when using the web or fat app interfaces that upload to the
JSS database first. Honestly I think the python-jss code is very close
and just needs to be worked on for issue 5.

On 11/26/14 3:09 PM, Scott Blake wrote:

Just yesterday, my TAM sent me an alternative workflow for adding
packages to a JDS:

1. Create the package in the database using the API.

2. The easiest method is to perform a GET on an existing package, save
   the XML and modify it. I attached it to the email.

3. Update the XML to include the correct name and display name for the
   package (case sensitive). You can supply a valid category or set
   other package preferences in the XML, or the changes can be made
   from Casper Admin or the web app after posting the XML
   4.

   The post is done with a command similar to the following:
   |curl -k -v -u username:password
   https://JSSURL.com:8443/JSSResource/packages/name/PackageName -T
   "/path/to/packages.xml" -X POST|

   Copy the package to /Library/JDS/shares/CasperShare/ (mac location,
   different on linux)

4. Update the JDS inventory by running 'sudo jamfds inventory'

—
Reply to this email directly or view it on GitHub
#174 (comment).

Everette Gray Allen Systems Programmer II
Office of Information Technology Macintosh Support Specialist
2620 Hillsborough St, Campus Box 7109
Raleigh, NC 27695-7109 AIM: EveretteAlln
919-515-4558 Everette_Allen@ncsu.edu

[eahrold]

@everetteallen You're well beyond the scope of my knowledge of JSS. Thanks for all that info.

Do you have any specific suggestions as to how we should show a "representative UI for the JSS/JDS sync"?

The 404 Error that early on seems strange, It's actually more like it's not even pointing to the correct API url for the distributionpoints.

When you do this does it look right?

defaults read com.github.autopkg JSS_URL

[everetteallen]

To be clear on the need here in Casper 9.x:
Package type files i.e. pkg and dmg files are loaded into the JSS database (downloadable_file_chunk_data) and pushed by a JSS process to the master JDS which then runs another process to sync to it's subordinates (and so on).
Script type files (sh, pl, js, etc) are loaded into the JSS database and stay only in the JSS database.
This is very different from Casper 8.x- behavior. Also the package (only not scripts) behavior for Casper 9 with File Share Distribution Points (i.e. not JDS) is different since in this case the files are copied directly to the shares but are not synced to other shares automatically instead the Casper Admin.app fat app is required to sync manually (or some other mechanism like rsync).
Folks are likely to have JDS moving forward.

[MScottBlake]

I haven't tried that approach yet, but the jamfds inventory is what checks
for necessary files. I would suspect that the other JDS servers (and even
the master, if not run) would sync up withink 5 minutes whenever they are
scheduled to run that command.

It is my understanding that this is the only way (short of direct sql) of adding a package to a JDS. If the package is not in the DB when it runs inventory, it will remove the package.

[everetteallen]

@eahrold in the case as of a Casper 9 JSS with JDS distribution points the confusion is that the JSS and JDS function as 1 unit even tho they are separate physical servers with different urls. So my JSS_URL and my JDS_Repos entry should and do have the same url. On the 404 I believe this may have been because of an earlier mis-match of the 2 entries so I will be testing (test server really is 404 right now).
For the UI I would suggest a check box under the API User Name widget entitled "Use JDS Master Distribution Point". This would require code to correctly setup the JSS_REPOS "preferences" automatically. As a note I would not disable the Distribution Point table in this case as it is possible to have both a JDS Master as the Primary Distribution point and to have additional AFP/SMB/WEBDAV file shares. A great value add for AutoPKGR would be to correctly upload the packages to all 4 of these types.

[everetteallen]

@MitchelSBlake your mileage may vary but my understanding is that the JSS sync to the master is the key that also trips off the JDS Master to sync to other JDS subordinates and the behavior I have observed is that just putting a package in the correct folder in the master JDS is not enough. In fact I have faked this by uploading a very small, dummy package of the same name to the JSS, let that sync and then replaced the one on the Master JDS and updated its checksums with still no joy.
Would be great to know if you different in your testing