Skip to content

[BUG]: Hidden exception in Finalize Job. Only visible with system.debug #5173

Open
@dieter-adesso

Description

@dieter-adesso

What happened?

During investigation of another agent issue, we activated system.debug & agent.diagnostics in Azure DevOps Pipelines.

Found randomly following exception in last phase of job execution in raw-logs and only with system.debug activated (not visible in normal logs):
##[debug]Error message: System.UnauthorizedAccessException: Access to the path '/var/log/azure/Microsoft.VisualStudio.Services.TeamServicesAgentLinux/events' is denied. ---> System.IO.IOException: Permission denied ...

I see such exception for many jobs, so it's not a one-time issue, but since it is only visible during system.debug, it's probably good to report to you.
Interestingly, the Pipeline works properly as if there was no error at all.

Versions

4.254.0 on Linux Ubuntu 22.04 LTS

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 22.04 LTS

Version controll system

No response

Relevant log output

2025-04-10T16:02:10.6506602Z ##[section]Finishing: Stop Containers
2025-04-10T16:02:10.6527141Z ##[section]Starting: Finalize Job
2025-04-10T16:02:10.6534632Z Cleaning up task key
2025-04-10T16:02:10.6535753Z Start cleaning up orphan processes.
2025-04-10T16:02:10.6747442Z ##[section]Finishing: Finalize Job
2025-04-10T16:02:10.6774291Z ##[debug]Starting diagnostic file upload.
2025-04-10T16:02:10.6774453Z ##[debug]Setting up diagnostic log folders.
2025-04-10T16:02:10.6775638Z ##[debug]Creating diagnostic log files folder.
2025-04-10T16:02:10.6776362Z ##[debug]Creating diagnostic log environment file.
2025-04-10T16:02:10.7532957Z ##[debug]Creating capabilities file.
2025-04-10T16:02:10.7550170Z ##[debug]Copying 2 worker diag logs from /agent/_diag.
2025-04-10T16:02:10.7561511Z ##[debug]Copying 1 agent diag logs from /agent/_diag.
2025-04-10T16:02:10.7564041Z ##[debug]Dumping of waagent.conf file
2025-04-10T16:02:10.7566298Z ##[debug]Dumping waagent.conf file is completed.
2025-04-10T16:02:10.7566429Z ##[debug]Dumping cloud-init logs.
2025-04-10T16:02:10.9465872Z ##[debug]This command must be run as root.

2025-04-10T16:02:10.9466239Z ##[debug]Cloud-init logs were not found.
2025-04-10T16:02:10.9466359Z ##[debug]Dumping cloud-init logs is ended.
2025-04-10T16:02:10.9466689Z ##[debug]The platform is not based on Debian - skipping debsums check.
2025-04-10T16:02:10.9466825Z ##[debug]Starting dumping Agent Azure VM extension logs.
2025-04-10T16:02:10.9471381Z ##[debug]Path to agent extension logs: /var/log/azure/Microsoft.VisualStudio.Services.TeamServicesAgentLinux
2025-04-10T16:02:10.9471721Z ##[debug]Archiving agent extension logs to: /agent/_diag/AgentLinuxExtensionLogs-20250410-160142-utc.zip
2025-04-10T16:02:10.9513823Z ##[debug]Failed to dump Agent Azure VM extension logs. Skipping.
2025-04-10T16:02:10.9518421Z ##[debug]Error message: System.UnauthorizedAccessException: Access to the path '/var/log/azure/Microsoft.VisualStudio.Services.TeamServicesAgentLinux/events' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.Enumeration.FileSystemEnumerator`1.Init()
   at System.IO.Enumeration.FileSystemEnumerable`1..ctor(String directory, FindTransform transform, EnumerationOptions options, Boolean isNormalized)
   at System.IO.Enumeration.FileSystemEnumerableFactory.UserEntries(String directory, String expression, EnumerationOptions options)
   at System.IO.Directory.InternalEnumeratePaths(String path, String searchPattern, SearchTarget searchTarget, EnumerationOptions options)
   at System.IO.ArchivingUtils.IsDirEmpty(String directoryFullName)
   at System.IO.Compression.ZipFile.CreateZipArchiveFromDirectory(String sourceDirectoryName, ZipArchive archive, Nullable`1 compressionLevel, Boolean includeBaseDirectory)
   at System.IO.Compression.ZipFile.DoCreateFromDirectory(String sourceDirectoryName, String destinationArchiveFileName, Nullable`1 compressionLevel, Boolean includeBaseDirectory, Encoding entryNameEncoding)
   at System.IO.Compression.ZipFile.CreateFromDirectory(String sourceDirectoryName, String destinationArchiveFileName)
   at Microsoft.VisualStudio.Services.Agent.Worker.DiagnosticLogManager.DumpAgentExtensionLogs(IExecutionContext executionContext, String supportFilesFolder, DateTime jobStartTimeUtc) in /mnt/vss/_work/1/s/src/Agent.Worker/DiagnosticLogManager.cs:line 325
   at Microsoft.VisualStudio.Services.Agent.Worker.DiagnosticLogManager.UploadDiagnosticLogsAsync(IExecutionContext executionContext, AgentJobRequestMessage message, DateTime jobStartTimeUtc) in /mnt/vss/_work/1/s/src/Agent.Worker/DiagnosticLogManager.cs:line 226
2025-04-10T16:02:10.9519703Z ##[debug]Zipping diagnostic files.
2025-04-10T16:02:10.9623623Z ##[debug]Uploading diagnostic metadata file.
2025-04-10T16:02:10.9641860Z ##[debug]Diagnostic file upload complete.
2025-04-10T16:02:10.9655196Z ##[section]Finishing: Deploy - DEV

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions