Clear the work folder after each build?

[jsauvexamarin]

Agent version and platform

2.11.0


OS X 10.11.6

VSTS type and version

VisualStudio.com


devops-mobile.visualstudio.com

What's not working?

I'd like to know if there'a a way to clear the _work folder after each build. My MacInCloud build server only has 100GB of drive space. With the OS and all the dev tools, I'm at 40GB free. I burn through that 40GB by doing builds in under 24 hours. I run 8 VSTS agents on that machine, and the _work folder of each spawns around 8 subfolders over time. I have Build.Clean set to all in my build definition variables, but the _work folders still remain very full, filling up my disk in very little time.

Is there some way to command the agent to clear its work folder after each build?

[ericsciple]

it's not built-in today (it also prevents investigating the layout is required after the build finishes). a workaround would be to add an always-run task to delete the build directory at the end of your build

[jsauvexamarin]

Cool, thanks @ericsciple. @madhurig is also checking into something for me, but I think this may be the most simple answer. I don't want to give up the build parallelism I have with 8 agents, so deleting the build folder with a task instead of scaling back my number of agents sounds like its worthwhile.

[ericsciple]

@bryanmacfarlane @chrisrpatterson for consideration

my personal preference is don't add this as a feature. user can add always-run task to best-effort delete agent.builddirectory. i think if we do add it, then it needs to be a separate option to also cleanup at the end of the build, otherwise it seems like we would get complaints the other way - i.e. build failed yet files not left for investigation

[bryanmacfarlane]

Agreed with @ericsciple

[jsauvexamarin]

My final solution was to manually delete the source and build folders after the build completes by using the "Delete Files" utility task. The $(Agent.BuildDirectory) and $(Build.SourcesDirectory) variables were useful for this.

[vikingsunil]

In my case, the deletion of $(Agent.BuildDirectory) is failing as the folder seems to be in use. The error message is ##[error]Error: Failed rmRF: rm: could not remove directory (code EBUSY):
Did anyone else face this problem? Is there a ready made solution for this?

[TingluoHuang]

@vikingsunil do you have file opened within $(agent.builddirectory) when you try to delete the folder?

[Mardoxx]

A way for the agent to clear the work directory after build automatically would be very welcomed....

@vikingsunil I have the same issue..

[bergmeister]

@ericsciple This feature is really needed. As a user who wants to run the agent also on a cheap azure vm on the d drive, I do not want to add cleanup steps to all my builds just because of that. This needs to be a configuration of the agent itself.

[lijunle]

I don't understand the reason to reject this feature. A simple search in the Internet will get lots of similar questions/issues. This is a big gap between product feature and user requirements. Why not make an out-of-box option-in feature? Then, users do not need to write various error-prone scripts/tools.

[phil123456]

our prod frontend crashed becos of this, ...the factg that vstsagent on linux does not clear its internal guts, is just unbelievable and should be sonsidered as a design flaw if not a bug

[4c74356b41]

@ericsciple this is indeed a ridiculous gap. why should i need to even think about it and face having to understand why can I delete in the agent dir and what I cant.

[bryanmacfarlane]

Just to be clear, there is a feature to clean the work directories such as the users sources and artifacts.

You can set build.clean to sources, binaries or all

However the feature does it before the next build for the reasons Eric laid out (workspace available for investigations). It would also be a waste for the vast majority of folks who use hosted and the machine is thrown away after the build.

Regarding avoiding understanding the "internal guts" of the agent, that's why we have variables to avoid you having to understand the layout of the work directory (and allows it to change). Use $(Agent.BuildDirectory) and $(Build.SourcesDirectory). That's also why we offer a delete task right out of the box. Drag it on and it does it for you.

If you use one of our hosted build queues, then you get a completely clean machine.

The ask in this issue was to also offer a clean option for after the build. I just wanted to clarify that the there are clean features and options so there's not a huge gap. It would be a variation on the existing feature.

[4c74356b41]

ok, after additional investigation, it appears you are wrong @bryanmacfarlane
https://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=vsts&tabs=schema#job

[4c74356b41]

@bryanmacfarlane @ericsciple one question though, i dont see any indication that the workspace was cleaned out, how do I verify that.

[bryanmacfarlane]

I was referring to a variable you can set on the variables tab in the UI. As you pointed out you can also set in yaml. Setting that in the yaml culminates in setting that variable. The variable works if you're doing a UI defined definition.

[4c74356b41]

@bryanmacfarlane what about the other question?

[EugenMayer]

We have implemented #1895 (comment) / https://github.com/EugenMayer/azure-agent-self-hosted-toolkit to deal with that issue on an agent-level (no realying on any pipeline annotation of step to cleanup).

Ensures every job runs in a clean workdir similar to microsofts hosted agent