Skip to content

Extended build failure logging#13705

Merged
PawelWMS merged 28 commits into3.0-devfrom
pawelwi/build_logging
May 15, 2025
Merged

Extended build failure logging#13705
PawelWMS merged 28 commits into3.0-devfrom
pawelwi/build_logging

Conversation

@PawelWMS
Copy link
Copy Markdown
Contributor

@PawelWMS PawelWMS commented May 7, 2025

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

  • The toolchain has been rebuilt successfully (or no changes were made to it)
  • The toolchain/worker package manifests are up-to-date
  • Any updated packages successfully build (or no packages were changed)
  • Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
  • Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
  • All package sources are available
  • cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
  • LICENSE-MAP files are up-to-date (./SPECS/LICENSES-AND-NOTICES/data/licenses.json, ./SPECS/LICENSES-AND-NOTICES/LICENSES-MAP.md, ./SPECS/LICENSES-AND-NOTICES/LICENSE-EXCEPTIONS.PHOTON)
  • All source files have up-to-date hashes in the *.signatures.json files
  • sudo make go-tidy-all and sudo make go-test-coverage pass
  • Documentation has been updated to match any changes to the build system
  • Ready to merge

Summary

Occasionally we get hit by build failures even though the build summary indicates that all packages built without issues and no specific details to the reason why the build failed are displayed. These failures seem to be random and hard to reproduce.

This change adds additional logging to be printed for such cases to help identify any dangling, unresolved nodes in the build graph, which may help us debug this issue.

Sample outputs:

  • Printing all nodes, even duplicates.
    image

  • Skipping duplicates.
    image

Does this affect the toolchain?

No.

Associated issues
Test Methodology
  • Local builds.
  • PR checks running the new unit tests.

@PawelWMS PawelWMS requested a review from a team as a code owner May 7, 2025 22:29
@microsoft-github-policy-service microsoft-github-policy-service Bot added Tools 3.0-dev PRs Destined for AzureLinux 3.0 labels May 7, 2025
Copy link
Copy Markdown
Member

@mfrw mfrw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we please add some documentation on how to use this :)

Comment thread toolkit/tools/internal/pkggraph/graphprinter.go Outdated
@eric-desrochers eric-desrochers requested a review from dmcilvaney May 8, 2025 13:15
Comment thread toolkit/tools/internal/pkggraph/graphprinter.go Outdated
@dmcilvaney dmcilvaney requested a review from Copilot May 8, 2025 17:52
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR extends the build failure logging to help diagnose sporadic build failures by printing hidden build blockers and refactoring unresolved dependency computations.

  • Added a new function PrintHiddenBuildBlockers in printresults.go to log potentially hidden blockers when no obvious failures are present.
  • Refactored the unresolved dependencies computation by extracting a new helper getUnresolvedDependencies.
  • Updated the fatal error message in scheduler.go to include the build log file for augmented troubleshooting.

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
toolkit/tools/scheduler/schedulerutils/printresults.go Added logging of hidden build blockers and refactored unresolved dependencies handling.
toolkit/tools/scheduler/schedulerutils/depsolver.go Added a helper to build a subgraph of blocked nodes to support the new logging.
toolkit/tools/scheduler/scheduler.go Updated fatal error messaging and added call to PrintHiddenBuildBlockers.
toolkit/tools/internal/pkggraph/pkggraph_test.go Extended test coverage for new unresolved nodes functionality.
toolkit/tools/internal/pkggraph/pkggraph.go Added AllUnresolvedNodes helper function.
toolkit/tools/internal/pkggraph/graphprinter*.go New and updated files for graph printing functionality.
Comments suppressed due to low confidence (2)

toolkit/tools/scheduler/schedulerutils/printresults.go:162

  • [nitpick] Consider including the goal node identifier or additional context in this log message to aid in debugging hidden blockers when the build unexpectedly fails.
logger.Log.Errorf("Detected a blocked build despite no obvious failures.")

toolkit/tools/scheduler/scheduler.go:228

  • [nitpick] Ensure that the fatal error message format remains consistent with other modules, and verify that the inclusion of the log file provides sufficient context for troubleshooting.
logger.Log.Fatalf("Unable to build package graph.\nFor details see the build summary section above and the build log '%s'.\nError: %s.", *logFlags.LogFile, err)

@PawelWMS PawelWMS force-pushed the pawelwi/build_logging branch from 2573adf to 2e61194 Compare May 9, 2025 16:29
@PawelWMS PawelWMS merged commit 8a67fc4 into 3.0-dev May 15, 2025
17 checks passed
@PawelWMS PawelWMS deleted the pawelwi/build_logging branch May 15, 2025 20:35
rlmenge added a commit to rlmenge/CBL-Mariner that referenced this pull request May 23, 2025
AkarshHCL pushed a commit to AkarshHCL/azurelinux that referenced this pull request Jul 31, 2025
SumitJenaHCL pushed a commit to SumitJenaHCL/azurelinux that referenced this pull request Aug 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.0-dev PRs Destined for AzureLinux 3.0 Tools

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants