Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache persist doesn't work on server 2012 R2 or 2016 (I assume) because it doesn't inclued tar #2595

Open
jabbera opened this issue Nov 12, 2019 · 22 comments
Assignees

Comments

@jabbera
Copy link

@jabbera jabbera commented Nov 12, 2019

Having issue with Tasks?

Since this is the embedded cache task I believe this belongs here.

Agent Version and Platform

Version of your agent? 2.160 on prem

OS of the machine running the agent? Windows 2012 R2

Azure DevOps Type and Version

dev.azure.com

https://dev.azure.com/gmollc

What's not working?

CacheBeta@1 isn't saving the cache because tar isn't included in 2012 R2.

Agent and Worker's Diagnostic Logs

2019-11-12T21:42:35.5033236Z ##[debug]Starting 'tar' with arguments '-h -v -cf "4092ddf38c57400cbae3ca85f9ae5178_archive.tar" -C "D:\SharedBuild2\1\s\.pipeline-cache\nuget" .'...
2019-11-12T21:42:36.0435164Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 8a46dbca-e561-4dd7-ab21-92c8c86e4de1
2019-11-12T21:42:36.0512117Z ##[error]The system cannot find the file specified
2019-11-12T21:42:36.0520742Z ##[debug]Processed: ##vso[task.logissue type=error;]The system cannot find the file specified
2019-11-12T21:42:36.0521458Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2019-11-12T21:42:36.1115284Z ##[debug]   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.ArchiveFilesToTarAsync(AgentTaskPluginExecutionContext context, String inputPath, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.GetUploadPathAsync(ContentFormat contentFormat, AgentTaskPluginExecutionContext context, String path, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.UploadAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, String path, CancellationToken cancellationToken, ContentFormat contentFormat)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, Func`1 restoreKeysGenerator, String path, CancellationToken token)
   at Agent.Plugins.PipelineCache.PipelineCacheTaskPluginBase.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
2019-11-12T21:42:36.1219290Z ##[section]Finishing: Cache NuGet packages
@jabbera jabbera changed the title Cache task doesn't work if you don't have tar in your path (I think) Cache persist doesn't work if you don't have tar in your path (I think) Nov 12, 2019
@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

You people are going to think I'm crazy, but I'm not... Here a screenshot of the procmon trace looking for tar.exe:

image

Here is a screenshot of it NEVER looking for 7z.exe from the same trace (The trace includes process start to terminate):

image

Just to prove I'm extra, extra not crazy:

image

And I've rebooted since to be sure the path propagated.

The ONLY way I can see procmon not reporting at least a search for 7z.exe is if somehow isWindows is false here:

if (isWindows && CheckIf7ZExists())

but I don't see how that is possible!?!?!?!

@johnterickson

This comment has been minimized.

Copy link
Contributor

@johnterickson johnterickson commented Nov 13, 2019

@fadnavistanmay Can you take a look?

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

Just to add more confusion to the mix. I provisioned a new 2019 server and everthing works (same pipeline). I'm going to try a 2012 R2 new. Then I'm going to put a new agent on my non-working 2012 r2. Will update this post as I investigate.

@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Nov 13, 2019

Hi @jabbera - Glad you were able to make it work on the 2019 server. Just a quick note, the error you are getting is in the SaveCache task, which always use tar; the code snippet you have shared is during the Restore Cache task, where we look for 7z and use that if present.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

@fadnavistanmay you hit the nail on the head. There is no fallback to 7-zip here.

ProcessStartInfo processStartInfo = GetCreateTarProcessInfo(context, archiveFileName, inputPath);
.

Server 2019 includes tar out of the box! Are you going to support windows os versions that don't include TAR for saving cache artifacts? (Since you have it for downloading I would expect you to:-))

@jabbera jabbera changed the title Cache persist doesn't work if you don't have tar in your path (I think) Cache persist doesn't work on server 2012 R2 because it doesn't inclued tar Nov 13, 2019
@jabbera jabbera changed the title Cache persist doesn't work on server 2012 R2 because it doesn't inclued tar Cache persist doesn't work on server 2012 R2 or 2016 (I assume) because it doesn't inclued tar Nov 13, 2019
@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Nov 13, 2019

@jabbera - did you explicitly installed tar on those machines, and it still fails?? You are using self-hosted agents, yes?

For now, you can fall back to old behavior by setting the variable AZP_CACHING_CONTENT_FORMAT to Files.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

@fadnavistanmay where from? Isn't that a 2019 only feature? (Unless you are talking about the mingw/gnuwin32 tar?) Additionally why would you have the 7 zip fallback for downloading caches but not uploading them? It doesn't make much sense to me.

Thanks for the workaround. I'll give it a try.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

@fadnavistanmay For whatever reason the environment variable doesn't work either. I took a surface level look at the code and it's not obvious why it's not working. I'm happy to setup a screen share or something if you want to take a deeper look. Here are snippits from my log:

[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'ALLUSERSPROFILE': 'C:\ProgramData'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'APPDATA': 'C:\Users\<SNIP>\AppData\Roaming'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'AZP_CACHING_CONTENT_FORMAT': 'Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramFiles': 'C:\Program Files\Common Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramFiles(x86)': 'C:\Program Files (x86)\Common Files'
[2019-11-13 20:27:16Z INFO EnvironmentCapabilitiesProvider] Adding 'CommonProgramW6432': 'C:\Program Files\Common Files'
2019-11-13T20:38:27.5300788Z ##[debug]Processed: ##vso[telemetry.publish area=AzurePipelinesAgent;feature=PipelineCache]{"FileCount":"0","PlanId":"b8fde65d-718b-4f2c-8a48-b3ff33b7809f","JobId":"0ab14b9f-e499-56d5-97b1-fd98b70ea339","TaskInstanceId":"aa203359-c3b8-5e0f-0fa6-2cb7b8dcce2a","CacheResult":"Miss","ActionDurationMs":"892","ActionName":"PipelineCache.RestoreCache","ActionResult":"Success","AttemptNumber":"1","ItemCount":"0","Level":"ThirdParty","CreatedUtcNow":"2019-11-13T20:38:26.5686059Z","SentUtcNow":"2019-11-13T20:38:27.4717776Z","BaseAddress":"https://vsblobprodcus3.vsblob.visualstudio.com/Ad31b77fc-dfbf-4068-8419-a4cc92bbfac6/","X_TFS_Session":"2ebdd9b2-d83c-46a7-836f-1080713ffccc","DeploymentEnvironment":"PRODUCTION","DeploymentEnvironmentIsProduction":"True","VSOAccount":"vsblobprodcus3","OSName":"Microsoft Windows","OSVersion":"6.3.9600","FrameworkDescription":".NET Core ","ProcessName":"Agent.PluginHost","DotNetReleaseDword":"-1","Version":"18.159.29324.0 built by: master (a2f0ba0f2a)","ExceptionCount":"0"}
2019-11-13T20:38:27.5302449Z ##[debug]Starting 'tar' with arguments '-h -v -cf "15fad05c549e42999f7259172fa315b6_archive.tar" -C "D:\SharedBuild1\2\s\.pipeline-cache\nuget" .'...
2019-11-13T20:38:28.1735655Z Information, ApplicationInsightsTelemetrySender correlated 1 events with X-TFS-Session 2ebdd9b2-d83c-46a7-836f-1080713ffccc
2019-11-13T20:38:28.1814906Z ##[error]The system cannot find the file specified
2019-11-13T20:38:28.1824931Z ##[debug]Processed: ##vso[task.logissue type=error;]The system cannot find the file specified
2019-11-13T20:38:28.1825655Z ##[debug]Processed: ##vso[task.complete result=Failed;]
2019-11-13T20:38:28.1988197Z ##[debug]   at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.RunProcessAsync(AgentTaskPluginExecutionContext context, ProcessStartInfo processStartInfo, Func`3 additionalTaskToExecuteWhilstRunningProcess, Action actionOnFailure, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.TarUtils.ArchiveFilesToTarAsync(AgentTaskPluginExecutionContext context, String inputPath, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.GetUploadPathAsync(ContentFormat contentFormat, AgentTaskPluginExecutionContext context, String path, CancellationToken cancellationToken)
   at Agent.Plugins.PipelineCache.PipelineCacheServer.UploadAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, String path, CancellationToken cancellationToken, ContentFormat contentFormat)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.ProcessCommandInternalAsync(AgentTaskPluginExecutionContext context, Fingerprint fingerprint, Func`1 restoreKeysGenerator, String path, CancellationToken token)
   at Agent.Plugins.PipelineCache.PipelineCacheTaskPluginBase.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.Plugins.PipelineCache.SavePipelineCacheV0.RunAsync(AgentTaskPluginExecutionContext context, CancellationToken token)
   at Agent.PluginHost.Program.Main(String[] args)
2019-11-13T20:38:28.2093865Z ##[section]Finishing: Cache NuGet packages
@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Nov 13, 2019

Hi @jabbera - The reason we went for tarring for upload was, the performance was almost the same for creating a "tar" - for both the tar and 7z process, but while downloading (untarring) - 7z gives much better performance.

Could you please tell us how/where are you setting the environment variable. And if you could give us the redacted logs of the entire build, that would be helpful.

Will it be possible for you. to give @johnterickson and me , permissions for https://dev.azure.com/gmollc , to investigate?

Thanks.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

I’m setting the variable as a system variable.

I’ll attach the logs in the AM.

Feel free to access the org.

This behavior is still crazy. It’s basically saying I can’t successfully complete a build that is a cache miss (because this terminates the build) from a 2012 [R2] or 2016 server unless I install a third party tar or set this environment variable(even if it worked). This cache task should be self contained and bullet proof if it can cause the build to fail. IMO considering you took a dependency on 7 zip for untaring a tar (on 2012 and 2016 since tar isn’t a default install) you should use it for creating one instead of failing.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 13, 2019

PS: the reason the tar thing doesn’t really work well is that tar is not a default option on 2016 or 2012. IMO: You should fall back on 7 zip, and if that isn’t there fall back on files all automatically unless you don’t plan on supporting those platforms as build platforms.

The other thing to keep in mind is if you run your build (and cache) artifacts from a machine that supports tar, but you want to use those artifacts on a machine that doesn’t (windows 2016 without 7 zip) the cache will not hydrate. This may be a use case you don’t care about but again, consistency across platforms would solve this issue. Any way you can pack a tar in the task or in the plugin or make the server deal with compatibility?

@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Nov 14, 2019

Hi @jabbera - I still can't access the org. Could you please check.

Thanks.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 14, 2019

I thought you meant on the backend. We don’t allow guest accounts in our tenant. I can open a ticket if that gives you more ability to get in. Otherwise I can do a screen share with you.

@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Nov 14, 2019

My bad. Let's check the logs first, if nothing significant comes up, we can do a screen share.

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 14, 2019

@fadnavistanmay I was able to reproduce this on my public azure devops. It really is as simple as installing the agent on a 2012 r2 server. I just created one in azure: https://dev.azure.com/mike-barry/Demo2012Issue/_build/results?buildId=420&view=results

If you need additional access please let me know

@jabbera

This comment has been minimized.

Copy link
Author

@jabbera jabbera commented Nov 18, 2019

@fadnavistanmay any chance to look at this in my public repo?

@johnterickson

This comment has been minimized.

Copy link
Contributor

@johnterickson johnterickson commented Nov 18, 2019

Let's make this error clearer that TAR needs to be on the path

@garfbradaz

This comment has been minimized.

Copy link

@garfbradaz garfbradaz commented Jan 8, 2020

I had this issue on Windows Server 2016. I had to install 7zip and gnuwin32 tar manually and add to the PATH. I'm running self hosted agents as a service, so I restarted the service after adding to the PATH - this fixed the problem for me (saving and restoring from the cache).

@johnterickson / @jtpetty Will the documentation be updated to include the hard dependency on these applications because at the moment there isn't anything:

https://docs.microsoft.com/en-us/azure/devops/pipelines/caching/index?view=azure-devops

I agree with @jabbera though, should this task really have these hard dependencies at all or should it install these applications within the tools cache within $(Agent.ToolsDirectory)

@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Jan 8, 2020

Hi @garfbradaz , sorry for the delay in documentation update. We have a PR out, it looks like it haven't merged yet. I'll follow up to get the docs updated.

@johnterickson

This comment has been minimized.

Copy link
Contributor

@johnterickson johnterickson commented Jan 10, 2020

Docs are live: https://docs.microsoft.com/en-us/azure/devops/pipelines/caching/index?view=azure-devops#required-software-on-self-hosted-agent

Thanks @fadnavistanmay!

@fadnavistanmay Can we add C:\Program Files\Git\usr\bin to the search path for tar on Windows?

@fadnavistanmay

This comment has been minimized.

Copy link
Contributor

@fadnavistanmay fadnavistanmay commented Jan 10, 2020

Hi @johnterickson , @b-barthel already has a PR out for "Can we add C:\Program Files\Git\usr\bin to the search path for tar on Windows?"

PR: b-barthel@6e0d9b7

@garfbradaz

This comment has been minimized.

Copy link

@garfbradaz garfbradaz commented Jan 16, 2020

Hi @garfbradaz , sorry for the delay in documentation update. We have a PR out, it looks like it haven't merged yet. I'll follow up to get the docs updated.

Smashing thanks guys - top work as always.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.