Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update data factory pipeline to FOCUS exports #571

Merged
merged 23 commits into from
Jun 10, 2024

Conversation

MSBrett
Copy link
Contributor

@MSBrett MSBrett commented Jan 31, 2024

πŸ› οΈ Description

Adds support for the new FOCUS export format in Azure Cost Management.

  • Added dataset for manifest.json generated by Azure Cost Management
  • Modified ingestion and msexports datasets to use blobpath rather than file and folder names.
  • Modified ingestion pipeline & triggers to use manifest.json (better queueing)
  • Modified managed exports feature to create FOCUS exports
  • Modified ingestion ETL pipeline to convert FOCUS data
  • Modified backfill pipeline to use existing monthly export rather than creating a new export (new feature in Azure Cost Management).
  • Modified folder structure in ingestion container to include date hierarchy folders.
  • Fixed file upload script's handling of scope object

πŸ“‹ Checklist

πŸ”¬ How did you test this change?

  • 🀏 Lint tests
  • 🀞 PS -WhatIf / az validate
  • πŸ‘ Manually deployed + verified
  • πŸ’ͺ Unit tests
  • πŸ™Œ Integration tests

πŸ™‹β€β™€οΈ Do any of the following that apply?

  • 🚨 This is a breaking change.
  • 🀏 The change is less than 20 lines of code.

πŸ“‘ Did you update docs/changelog.md?

  • βœ… Yes (required for dev PRs)
  • ➑️ Will cover in a future PR (feature branch PRs only)
  • ❎ Not needed (small/internal change)

πŸ“– Did you update documentation?

  • βœ… Public docs in docs (required for dev)
  • βœ… Internal dev docs in src (required for dev)
  • ➑️ Will cover in a future PR (feature branch PRs only)
  • ❎ Not needed (small/internal change)

@MSBrett MSBrett added Status: πŸ”¬ Code review Issue is currently in code review with open PRs Area: Data factory Data Factory integration Needs: Review πŸ‘€ PR that is ready to be reviewed Solution: FinOps hubs Data pipeline solution labels Jan 31, 2024
@MSBrett MSBrett requested a review from flanakin January 31, 2024 15:39
@MSBrett MSBrett self-assigned this Jan 31, 2024
@MSBrett MSBrett marked this pull request as ready for review January 31, 2024 15:39
@flanakin flanakin added this to the 0.3 milestone Feb 3, 2024
src/templates/finops-hub/modules/hub.bicep Outdated Show resolved Hide resolved
src/templates/finops-hub/modules/dataFactory.bicep Outdated Show resolved Hide resolved
src/templates/finops-hub/modules/dataFactory.bicep Outdated Show resolved Hide resolved
src/templates/finops-hub/modules/dataFactory.bicep Outdated Show resolved Hide resolved
src/templates/finops-hub/modules/dataFactory.bicep Outdated Show resolved Hide resolved
@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs: Attention πŸ‘‹ Issue or PR needs to be reviewed by the author or it will be closed due to no activity label Feb 6, 2024
@flanakin flanakin removed the Needs: Attention πŸ‘‹ Issue or PR needs to be reviewed by the author or it will be closed due to no activity label Mar 13, 2024
@flanakin flanakin modified the milestones: 0.4, Hubs - Managed exports Apr 3, 2024
@microsoft-github-policy-service microsoft-github-policy-service bot added the Needs: Attention πŸ‘‹ Issue or PR needs to be reviewed by the author or it will be closed due to no activity label Apr 6, 2024
@MSBrett MSBrett removed the Needs: Attention πŸ‘‹ Issue or PR needs to be reviewed by the author or it will be closed due to no activity label May 20, 2024
@MSBrett MSBrett dismissed flanakin’s stale review May 20, 2024 14:27

Outdated after changes

@MSBrett MSBrett marked this pull request as draft May 20, 2024 14:27
@MSBrett MSBrett requested review from flanakin and removed request for flanakin May 20, 2024 14:27
@MSBrett MSBrett marked this pull request as ready for review May 20, 2024 14:27
Co-authored-by: Michael Flanakin <flanakin@users.noreply.github.com>
@flanakin flanakin added Status: ▢️ Ready Issue is ready for a dev to start work and removed Status: πŸ”¬ Code review Issue is currently in code review with open PRs Needs: Review πŸ‘€ PR that is ready to be reviewed labels Jun 5, 2024
@MSBrett MSBrett merged commit dd3e86d into features/exports Jun 10, 2024
1 check passed
@MSBrett MSBrett deleted the features/msbrett/exportsv2 branch June 10, 2024 15:53

@all-contributors please add MSBrett for code

Copy link
Contributor

@microsoft-github-policy-service[bot]

@MSBrett already contributed before to code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Data factory Data Factory integration Solution: FinOps hubs Data pipeline solution Status: ▢️ Ready Issue is ready for a dev to start work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants