Ensure new-certificate can work without a profile, or document how to load profiles

[PaulStovell]

Issue:

http://help.octopusdeploy.com/discussions/problems/13114-tentacle-new-certificate-throwing-unauthorisedaccessexception

Can we try and reproduce this, and then:

1. Look at fixing our certificate generation so that the whole thing just works, or
2. At least document how to force the profile to be loaded so that it can work

[nblumhardt]

While investigating this, have found that our check of LASTERROR is flawed in the same way as the AD integration's one:

Octopus Deploy: Tentacle version 1.0.0.0

-------------------------------------------------------------------------------
A fatal exception occurred
System.IO.FileNotFoundException: Could not load file or assembly 'Octopus.Shared.XmlSerializers' or one of its dependencies. The system cannot find the file specified.
File name: 'Octopus.Shared.XmlSerializers'
   at System.Reflection.RuntimeAssembly._nLoad(AssemblyName fileName, String codeBase, Evidence assemblySecurity, RuntimeAssembly locationHint, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean throwOnFileNotFound, Boolean forIntrospection, Boolean suppressSecurityChecks)
   at System.Reflection.RuntimeAssembly.LoadWithPartialNameInternal(AssemblyName an, Evidence securityEvidence, StackCrawlMark& stackMark)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at Octopus.Shared.Internals.CertificateGeneration.Win32ErrorHelper.ThrowExceptionIfGetLastErrorIsNotZero() in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Internals\CertificateGeneration\Win32ErrorHelper.cs:line 12
   at Octopus.Shared.Internals.CertificateGeneration.CryptContext.Open() in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Internals\CertificateGeneration\CryptContext.cs:line 37
   at Octopus.Shared.Security.CertificateGenerator.Generate(String fullName, Boolean exportable) in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Security\CertificateGenerator.cs:line 25
   at Octopus.Shared.Security.CertificateGenerator.GenerateNew(String fullName) in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Security\CertificateGenerator.cs:line 11
   at Octopus.Shared.Configuration.TentacleConfiguration.GenerateNewCertificate() in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Configuration\TentacleConfiguration.cs:line 158
   at Octopus.Tentacle.Commands.NewCertificateCommand.Start() in c:\Development\Octopus\OctopusDeploy\source\Octopus.Tentacle\Commands\NewCertificateCommand.cs:line 25
   at Octopus.Shared.Startup.AbstractCommand.Octopus.Shared.Startup.ICommand.Start(String[] commandLineArguments, ICommandRuntime commandRuntime) in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Startup\AbstractCommand.cs:line 42
   at Octopus.Shared.Startup.OctopusProgram.Start(ICommandRuntime commandRuntime) in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Startup\OctopusProgram.cs:line 162
   at Octopus.Shared.Startup.ConsoleHost.Run(Action`1 start, Action shutdown) in c:\Development\Octopus\OctopusDeploy\source\Octopus.Shared\Startup\ConsoleHost.cs:line 34

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

-------------------------------------------------------------------------------

(Edit: verified that this error is only produced with runas /noprofile, so making the assumption that it reflects the Access Denied error referenced in the post above.)

[nblumhardt]

It seems like to make this "just work" will require us to use LoadUserProfile() to load the current user's profile when generating the certificate (http://msdn.microsoft.com/en-us/library/windows/desktop/bb762281(v=vs.85).aspx). The docs on this are pretty unclear, I'm not confident we'll solve more problems than we introduce by trying to go this way.

The second option - documenting what to do - seems best right now. However, the solution is effectively:

runas /profile /user:<username> "C:\...\Tentacle.exe new-certificate"

As a test user, this succeeds.

BUT, if I specify my own username this fails on my machine because of the presence of a UAC requirement in Tentacle.exe.manifest:

RUNAS ERROR: Unable to run - C:\Development\Octopus\OctopusDeploy\source\Octopus.Tentacle\bin\Tentacle.exe new-certificate
740: The requested operation requires elevation.

I'm going to implement a "Knowledge Base" entry for the error, and I've raised #356 for the manifest issue hoping that can be cleared up easily.

[nblumhardt]

===============================================================================
Crypto functions require the Windows User Profile
-------------------------------------------------------------------------------
Various cryptographic functions used by Octopus Deploy require the Windows user profile to have
been loaded. Some remote administration scenarios run commmands in processes without user profile
information; to successfully run the problem command, invoke it from the command-line using RUNAS,
e.g.: `runas /profile /user:<username> "C:\...\Tentacle.exe new-certificate"`.
See: http://g.octopushq.com/CryptoRequiresUserProfile
===============================================================================

[nblumhardt]

See whether CRYPT_MACHINE_KEYSET (32) can be used to make this work without a profile.

[erichexter]

Would it be possible to generate the certificate on a different machine under a process where the profile is loaded and then install the cert on the machine?

[nblumhardt]

Hi Eric,

Yes, this can be done in theory - but hasn't been tested.

On the source machine:

• Create an X509Certificate2 - must have private key
• Export it and encode as base64 - encoded = Convert.ToBase64String(certificate.Export(X509ContentType.Pkcs12))

On the target machine:

• Protect using DPAPI:

protected = Convert.ToBase64String(
                        ProtectedData.Protect(
                            Encoding.UTF8.GetBytes(encoded),
                            null,
                            DataProtectionScope.LocalMachine))

• Write to Tentacle.config
• Restart Tentacle

Can't guarantee it will work but has a good chance.

[erichexter]

I think DPAPI is actually going to need the profile loaded. Which I cannot
force to happen in a remote ps session.

[nblumhardt]

Local machine scope should do it, if that's possible.

[erichexter]

I think this boils down to the same issue that was document in this post.
http://blog.stangroome.com/2012/05/17/powershell-remoting-user-profiles-delegation-and-dpapi/
.

maybe my entire approach to automating the tentacle installation is wrong.
What is the guidance for creating a machine and installing the tentacle?
I have started down the remote powershell route since that is supported
out of the box with win2012 and azure virtual machines. Should I use
another automation technology to push this and execute the installation?

[nblumhardt]

Ick, no good then. Thanks for the info.

I think PowerShell automation is the goal here, reopening this issue since we reverted the fix.

Here's a workaround you might consider:

• Rename Tentacle.exe.manifest to something else
• runas /profile /user:<username> "C:\...\Tentacle.exe new-certificate"
• Restore Tentacle.exe.manifest

If that does solve the problem for you I'll look at how we might build this in (or even just provide a non-elevating executable cut from Tentacle.exe to do the generation)

Thanks for the continued input.

[erichexter]

I spent the last two days trying variations of using start-process,
systemdiag,process, and runas .. since its scripted and runas prompts for
a password it makes things a little more difficult. I tried doing a

cmd /c "echo password | runas /profile /user: .......

But I was not able to get that to run and accept the password.

For now, I will have to pop up a rdp and let someone click on a batch file,
that i can push to the machine remotely.

Its too bad, I was able to install, team city, sql server, java, chrome,
and get a teamcity agent up without ever logging into the machine,. I hope
to get the same automation with the tentacle. It way more important for me
to get the production boxes totaly automated and creating the cert for the
tentacle is that only thing left on my list, so please let me know if there
is anything else I should try, I am available to help.

[nblumhardt]

Hi Eric, I'm going to look at doing this without loaded profiles, but in the meantime - deleting Tentacle.exe.manifest did resolve problems with the RUNAS command for me. I know it was pretty fragile though, not a great experience.

Will follow up here with our next attempt.

BTW, @robdmoore might have encountered this building https://github.com/MRCollective/AzureWebFarm.OctopusDeploy --- any insights Rob?

[robdmoore]

No, thankfully I didn't come across that problem, but that makes sense because I'm pretty sure that the RoleEntryPoint runs as a real user.

@erichexter feel free to check out https://github.com/MRCollective/AzureWebFarm.OctopusDeploy :)

[nblumhardt]

@robdmoore thanks Rob!

[erichexter]

Thanks I checked it out. Cool project!.

I think I will try running the configure as a scheduled task on the
machine. That may allow me to get a workaround in place.

On Wednesday, January 8, 2014, Nicholas Blumhardt wrote:

@robdmoore https://github.com/robdmoore thanks Rob!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-31896363
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[nblumhardt]

@erichexter just curious, is it possible to run runas /profile for anything on the target box? By the sounds of things, forcing the profile to load once should make it available to subsequent processes, i.e. could we do a dummy runas to force the profile then run Tentacle? I'm clutching at straws here of course.

Our other option seems to be to investigate a purely-managed API for cert generation, and possibly to bypass DPAPI when certs are configured this way. Trying to steer away from this because of the fragility that seems to come with all changes cryptographic.

[erichexter]

Let me see is I can get that to work with running dir or something else.

On Wednesday, January 8, 2014, Nicholas Blumhardt wrote:

@erichexter https://github.com/erichexter just curious, is it possible
to run runas /profile for anything on the target box? By the sounds of
things, forcing the profile to load once should make it available to
subsequent processes, i.e. could we do a dummy runas to force the profile
then run Tentacle? I'm clutching at straws here of course.

Our other option seems to be to investigate a purely-managed API for cert
generation, and possibly to bypass DPAPI when certs are configured this
way. Trying to steer away from this because of the fragility that seems to
come with all changes cryptographic.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-31896862
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[erichexter]

I was able to get this working by invoking the process as a scheduled task
which was created using a invoke-command remotely. This is the last piece
I needed to script the creation of entire environments of our application
in azure. I have some clean up of the code to do.. once I get it cleaned up
an sanitized I will blog it and post it here. Definitely not my first
choice for doing this, but I can live with the solution, for now.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

On Wed, Jan 8, 2014 at 8:18 PM, Eric Hexter eric.hexter@gmail.com wrote:

Let me see is I can get that to work with running dir or something else.

On Wednesday, January 8, 2014, Nicholas Blumhardt wrote:

@erichexter https://github.com/erichexter just curious, is it possible
to run runas /profile for anything on the target box? By the sounds of
things, forcing the profile to load once should make it available to
subsequent processes, i.e. could we do a dummy runas to force the
profile then run Tentacle? I'm clutching at straws here of course.

Our other option seems to be to investigate a purely-managed API for cert
generation, and possibly to bypass DPAPI when certs are configured this
way. Trying to steer away from this because of the fragility that seems to
come with all changes cryptographic.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-31896862
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[dtjensen]

Eric - I am having the exact same issue. Any help you can post on how to do this with the schedule task is greatly appreciated.

[erichexter]

I was going to try to reply with the code inline, but that will be a mess.
I will put it into a gist. but here is the basic steps.

use invoke-command to remotely create a scheduled task .
scheduled task runs on the machine.

1. downloads a bootstrap script
2. downloads msi
3. configurres the tentacle

[nblumhardt]

Investigating the option of generating the cert on another maching, then importing it into Tentacle. This seems like it will be fine with machine-scoped DPAPI; the only real question is whether we should require a password on the exported certificates, which adds a little bit of effort. Close to having this PoC'd but now off for a short holiday, so picking this up on my return.

[nblumhardt]

Done; to create a certificate on a workstation machine run:

Tentacle.exe new-certificate --export-file="cert.txt"

To import the certificate on the Tentacle machine, run:

Tentacle.exe import-certificate --from-file="cert.txt"

(The target Tentacle should probably not be running when this is done.)

The exported ceritificate format is a simple clear-text base-64 file; this should be protected/treated as sensitive configuration.

Any issues (once dropped in 2.4) please let us know. Thanks for all the input!

[niven01]

Hi, nblumhardt

I just tried Tentacle.exe new-certificate --export-file="cert.txt" and received the following error. Im using Tentacle version 2.1.3.1223

A fatal exception occurred
System.ArgumentException: Unrecognized command line arguments: --export-file=cert
.txt

I've also tried --import-file bu this is not recognised. Are these arguments deprecated in V2 and is there another way of achieving this.

My goal is to deploy tentacle via a Chef cookbook

[erichexter]

It will be in version 2.4

On Thursday, April 3, 2014, Nielsen Pierce notifications@github.com wrote:

Hi, nblumhardt

I just tried Tentacle.exe new-certificate --export-file="cert.txt" and
received the following error. Im using Tentacle version 2.1.3.1223

A fatal exception occurred
System.ArgumentException: Unrecognized command line arguments:
--export-file=cert
.txt

Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-39447732
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[erichexter]

How close it 2.4 getting to a preview release? I am looking into vagrant
for setting up environments and it just uses winrm the same way I was
invoking remote powershell so this would be really nice to put in place to
enable installing the tentacles

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

On Thu, Apr 3, 2014 at 10:02 AM, Eric Hexter eric.hexter@gmail.com wrote:

It will be in version 2.4

On Thursday, April 3, 2014, Nielsen Pierce notifications@github.com
wrote:

Hi, nblumhardt

I just tried Tentacle.exe new-certificate --export-file="cert.txt" and
received the following error. Im using Tentacle version 2.1.3.1223

A fatal exception occurred
System.ArgumentException: Unrecognized command line arguments:
--export-file=cert
.txt

Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-39447732
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[nblumhardt]

I could be optimistic, but it is looking like we'll have a -pre up within a week. HTH.

[nblumhardt]

^H^H^H^Ham known for always being optimistic ;)

[nblumhardt]

2.4.1 is in preview now.

[erichexter]

Great
On Wednesday, April 30, 2014, Nicholas Blumhardt notifications@github.com
wrote:

2.4.1 is in preview now.

Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-41867490
.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

[robdmoore]

Are you passing the --console command line argument?

[paulzah]

Hey! Sorry for the lag in response; that’s why I deleted the post :D . Looking to migrate from Red Gate Deployment Manager to Octopus and still have to get used to great documentation

Paul Zah
Release Engineer – Everybody’s special, in the same way
Skyscanner, Edinburgh EH3 9EN
skyscanner.nethttp://www.skyscanner.net/

From: Robert Moore [mailto:notifications@github.com]
Sent: 26 May 2014 12:08
To: OctopusDeploy/Issues
Cc: Paul Zah
Subject: Re: [Issues] Ensure new-certificate can work without a profile, or document how to load profiles (#353)

Are you passing the --console command line argument?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-44179396.

[paulzah]

Hello.

We are looking into migrating to Octopus from RGDM and I’m trying to write an automatic migration script. I’ve hit an issue I’ve been really struggling with for the couple of days (the only bit of documentationhttps://github.com/OctopusDeploy/OctopusDeploy-Api/wiki/DeploymentProcesses that was not easy-to-understand).

I am trying to add code-generated steps into a deployment process. The docs say that you can use a PUT request to /api/deploymentprocesses/{id}, but I have not been able to deduce the expected contents of the body of the request (I prefer API calls due to the nature of the Powershell migration script).

I would really appreciate any assistance you could provide with this.

Paul Zah
Release Engineer – Transforming a process into a decision
Skyscanner, Edinburgh EH3 9EN
skyscanner.nethttp://www.skyscanner.net/

From: Robert Moore [mailto:notifications@github.com]
Sent: 26 May 2014 12:08
To: OctopusDeploy/Issues
Cc: Paul Zah
Subject: Re: [Issues] Ensure new-certificate can work without a profile, or document how to load profiles (#353)

Are you passing the --console command line argument?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/353#issuecomment-44179396.

[nblumhardt]

Hi @paulzah - thanks for getting in touch; automatic migration down this path sounds like a tricky undertaking, though we may have a few ideas to share. Can you please contact support@octopusdeploy.com so that we can continue the discussion there? Many thanks!

[robdmoore]

Can I be included on that discussion? I'm really interested in the automated provision of OctopusDeploy steps...

[nblumhardt]

Hey Rob - the discussion will probably stick to how best to migrate from RGDM, so maybe not an exact match. Feel free to open another ticket with any Q's you have in that area though!

[robdmoore]

Cool. Thanks.

[Shogan]

I've sent a message in to the support email, but am also looking for a way to deploy the Octopus tentacle agent to an Azure VM using WinRM / PowerShell.

I have pretty much automated my entire environment setup in Azure, but its just the new-certificate part using Tentacle.exe that does not work without a profile. Have you guys got any workarounds or solutions to this?

[erichexter]

Yes, you need to run your tentacle registration as a scheduled task, on the
server. when you do this, the user profile will be available and the
certificate creation will work.

Here is gist of two functions I use to do this in azure using remote
powershell. https://gist.github.com/erichexter/b0cca2ff2e3ab120cec8

Feel free to ask questions on this thread or comment on the gist itself and
I can explain anything that is unclear.

Eric Hexter

blog | http://Hex.LosTechies.com
info | http://www.linkedin.com/in/erichexter

On Thu, Aug 14, 2014 at 10:18 AM, Sean notifications@github.com wrote:

I've sent a message in to the support email, but am also looking for a way
to deploy the Octopus tentacle agent to an Azure VM using WinRM /
PowerShell.

I have pretty much automated my entire environment setup in Azure, but its
just the new-certificate part using Tentacle.exe that does not work without
a profile. Have you guys got any workarounds or solutions to this?

Reply to this email directly or view it on GitHub
#353 (comment)
.

[Shogan]

Ahh, brilliant stuff Eric.

I will give this a go tonight or tomorrow - looks solid enough after having had a quick look at the gist. Thank you for your quick reply here 👍

[mwrock]

I just ran into this trying to provision a tentacle via chef. We perform the initial provisioning run over winrm which has all the same restrictions as powershell remoting. @erichexter's approach is unfortunately the only approach likely to work here. I had success installing via boxstarter.org which basically does just that. It would be great if there was another way of generating the cert without needing the local user token. Scheduled tasks are such a pain. Funny, I just wrote a blog last week on this exact topic: http://www.hurryupandwait.io/blog/safely-running-windows-automation-operations-that-typically-fail-over-winrm-or-powershell-remoting

[erichexter]

I agree this approach sucks. But it has been reliable, we have scripted
200+ installs in azure and it works.

On Tuesday, January 27, 2015, Matt Wrock notifications@github.com wrote:

[Tazer]

Has there been any updates to this ? Or the only solution is to run the configuration as a scheduled task ? @PaulStovell @erichexter

[erichexter]

There is a new export and import commands on the tentacle exe

On Saturday, September 26, 2015, Patrik notifications@github.com wrote:

[Tazer]

Thanks, that lead me to : http://docs.octopusdeploy.com/display/OD/Export+and+import+Tentacle+certificates+without+a+profile

So now it's working :)

[erichexter]

@mwrock do you have a confirmed way to configure via chef ?

[mwrock]

Unfortunately I don't have access to those recipes anymore @erichexter . I do remember we had to have the recipe create a scheduled task which ran a script that did the install. We then polled the tentacle endpoint waiting or it to come online.

[DevOpsFu]

I wrote a blog post on how we got around this using OpenSSL to pre-stage certificates for importing.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. If you think you've found a related issue, please contact our support team so we can triage your issue, and make sure it's handled appropriately.