Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[COST-5148] update EC2 compute insert sql #5206

Merged
merged 7 commits into from
Jul 9, 2024

Conversation

djnakabaale
Copy link
Contributor

@djnakabaale djnakabaale commented Jul 3, 2024

Jira Ticket

COST-5148

Description

This change will update sql inserting records into reporting_awscostentrylineitem_summary_by_ec2_compute table to:

  • filter out empty resource ids
  • offset savings from SavingsPlanCoveredUsage

Testing

  1. Checkout Branch

  2. Restart Koku

  3. Update yaml to include savings for EC2 similar to this

    generators:
    - EC2Generator:
        start_date: {{start_date}}
        end_date: {{end_date}}
        processor_arch: 64-bit
        resource_id: 55555555
        product_sku: VEAJHRNKTJZQ
        region: us-east-1a
        tags:
            resourceTags/user:version: prod
            resourceTags/user:dashed-key-on-aws: dashed-value
            resourceTags/user:Mapping: a1
            resourceTags/user:Map: a2
            resourceTags/user:Name: instance-name-1
        cost_category:
            costCategory/env: prod
        instance_type:
            inst_type: m5.large
            physical_cores: 1
            vcpu: 2
            memory: '8 GiB'
            storage: 'EBS Only'
            family: 'General Purpose'
            cost: 0.096
            rate: 0.096
            saving: 0.0026
    
  4. Load test data for AWS

  5. Launch postgres shell and check for total unblended_cost for a given source and month as shown below

    postgres=# select sum(unblended_cost) from reporting_awscostentrylineitem_summary_by_ec2_compute where usage_start = '2024-07-01' and source_uuid = '13dddb00-8e6c-4ea1-b504-fd95491104b4'::uuid;
        sum      
    --------------
    53.568000000
    (1 row)
    
    postgres=# 
    
  6. Verify that total unblended_cost againt trino results

    trino:org1234567> select sum(lineitem_unblendedcost) from aws_line_items_daily where year='2024' and month='07' and source='13dddb00-8e6c-4ea1-b504-fd95491104b4' and product_productfamily LIKE '%Compute Instance%' and lineitem_lineitemtype!='SavingsPlanCoveredUsage';
    _col0  
    --------
    53.568 
    (1 row)
    

Release Notes

  • proposed release note
* [COST-5148](https://issues.redhat.com/browse/COST-5148) EC2 summary filter out empty resource ids and SavingsPlanCoveredUsage entries

filter out empty resource ids
offset savings from SavingsPlanCoveredUsage
@djnakabaale djnakabaale self-assigned this Jul 3, 2024
@djnakabaale djnakabaale changed the title [COST-5148] update insert sql [COST-5148] update EC2 compute insert sql Jul 3, 2024
Copy link

codecov bot commented Jul 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.1%. Comparing base (12d5853) to head (32ee73f).

Additional details and impacted files
@@          Coverage Diff          @@
##            main   #5206   +/-   ##
=====================================
  Coverage   94.1%   94.1%           
=====================================
  Files        376     376           
  Lines      31262   31262           
  Branches    4603    4603           
=====================================
  Hits       29430   29430           
  Misses      1165    1165           
  Partials     667     667           

@djnakabaale djnakabaale added the aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests label Jul 8, 2024
@djnakabaale djnakabaale marked this pull request as ready for review July 8, 2024 18:41
@djnakabaale djnakabaale requested review from a team as code owners July 8, 2024 18:41
@djnakabaale djnakabaale enabled auto-merge (squash) July 9, 2024 14:39
@djnakabaale djnakabaale merged commit fcdf02d into main Jul 9, 2024
11 checks passed
@djnakabaale djnakabaale deleted the cost-5148-sql-offset-savings branch July 9, 2024 14:57
djnakabaale added a commit that referenced this pull request Jul 9, 2024
… entries (#5206)

* [COST-5148] update insert sql
filter out empty resource ids
offset savings from SavingsPlanCoveredUsage

* closing CASE statement

* clean up comment

* remove case stmts in favor of filtering out SavingsPlanCoveredUsage

* clean up
djnakabaale added a commit that referenced this pull request Jul 19, 2024
…5117)

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: insert new filters.

* feat: insert new filters and order by params.

* feat: customizing provider map and serializer.

* fix: changing TIME_CHOICES options.

* feat: first unit tests.

* feat: wip.

* feat: fixing provider map.

* feat: fixing provider map.

* update ec2 annotations to get all required fields

* use AWSEC2ComputeQueryParamSerializer

* feat: creating orderby and groupby serializers for ec2

* feat: unit tests for filters

* feat: removing group_by - not needed on ec2

* feat: fixing orderby serializer and starting units tests

* fix: typo

* feat: changing usage_hours to usage_amount

* feat: fixing unit test.

* wip: blocking some filters and unit test.

* feat: unit tests for group by filter

* flake8 fix

* feat: inserting more filters on validate function.

* feat: updating validate method to use similar logic and add filters.

* fix: changing unit tests for some filters.

* feat: testing filter combinations and flake8 checks.

* fix: test

* feat: serializer Unit tests and view Unit test fix

* flake8 fix

* fix: new approach to satisfy CodeCov

* fix:: getting rid of validate custom method.

* fix: commenting tags.

* fix: validarte functions, tests.

* handle filter params for specific report type

* transform tags to desired ui format

* default to monthly resolution on the EC2 endpoint

* add special pagination for EC2

* use default report type time period settings if exists

* fix typo

* fix: fixing parameters validations.

* [COST-5141] Fix management command to use continue instead of return. (#5173)

* [COST-5128] Process new subs tagging strategy to identify non-converted instances (#5162)

* [COST-4745] Add data_transfer_direction to OCP on GCP Trino tables (#5130)

* [COST-4741] Add data_transfer_direction for AWS network costs to Trino tables (#5129)

* [COST-5168] - Adding new penalty pipeline (#5176)

* [COST-5168] - Adding new penalty pipeline

* Improve our logging readability (#5178)

* add prometheus metrics for new queues (#5179)

* add v3.3.0 operator commits (#5143)

* [COST-5124] Improve Trino migration management command (#5163)

* Add exponential backoff and logging to retries
* Change log level to reflect severity
* Explicit SQL alias for clarity
* Catch and log exception instead of exiting
* Add return type hints
* Return if unsuccessful
  No point in verifying if the SQL did not run correctly
* Fine tune exponential backoff
* Create action class for adding/verifying columns were added
* Assign default list using default_fatory
  Instead of doing it in the post_init, which get’s a little weird.
* Add drop column action
* Quote items in logs for better legibility
* Consolidate action classes
  We lose some of the action-specific logging messages, but there is less
  code overall. I’m not sure how this scale to the action related to dropping.
* Change local variable name
  No need to add a prefix to differentiate it from the parameter name.
* Use a set to prevent running on the same schema multiple times

Co-authored-by: Cody Myers <codymyers63@gmail.com>

* Filter accounts by matching criteria during subs processing to prevent unnecessary SQL from running (#5184)

* Update tasks.py (#5185)

* clean up grafana dashboard (#5183)

* Skip OCPCloud tag SQL if key is present in cache but value is None (#5186)

* [COST-5196] - Send OCP tasks to correct queues (#5187)

* [COST-5196] - Send OCP tasks to correct queues

* [COST-5176] correctly pass context dictionary within log_json function call (#5182)

* [COST-5176] correctly pass context dictionary within log_json function call

* add unittests for exceptions in generate_report

* batch delete S3 files (#5180)

* Bump urllib3 from 1.26.18 to 1.26.19 in the pip group across 1 directory (#5172)

Bumps the pip group with 1 update in the / directory: [urllib3](https://github.com/urllib3/urllib3).


Updates `urllib3` from 1.26.18 to 1.26.19
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/1.26.19/CHANGES.rst)
- [Commits](urllib3/urllib3@1.26.18...1.26.19)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: indirect
  dependency-group: pip
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add flower as a dev dependency (#5189)

* Add docs

* [COST-4844] Serializer update for ordering by storageclass (#5174)

* Switch to using podman in build_deploy (#5193)

The VM used in CI is now RHEL 8

* skip polling providers still processing (#5181)

* skip polling providers that are still processing

* [COST-5214] Move TARGETARCH declaration to the top of the Dockerfile (#5195)

There is a bug in podman where this is only used correctly for the
multi-stage build if it is defined as the first line.

Update Jenkinsfile to use RHEL 8

Unfortunately this breaks the image build for Docker. I'll fix that in a followup PR.

* [COST-5213] - fix S3 prepare (#5194)

* Switch default parquet flag to prevent iterating on all files in each worker when there is nothing to delete

* [COST-5214] pass build-arg to docker build command (#5196)

* [COST-5216] Delete filtering optimization (#5197)

* Revert "[COST-5216] Delete filtering optimization (#5197)" (#5200)

This reverts commit 97ba98e.

* [COST-5226] - Skip S3 delete (daily flow) if we have marked deletion complete. (#5198)

* dont attempt more S3 deletes if we have marked deletion complete

* [COST-5076] upgrade to python 3.11 (#4444)

* upgrade to python 3.11

* pipfile update

* add gcc-c++ compiler

Co-authored-by: Sam Doran <sdoran@redhat.com>

* update test

* replace gcc with gcc-c++

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>

* [COST-5228] log outside for loop (#5202)

* [COST-5228] log outside for loop

* additional log clean up

* add context to logs in _remove_expired_data func

* log s3 batch deletes (#5204)

* log s3 batch deletes

* [COST-5219] Correctly report VM usage for metering when billing record is split (#5201)

* [COST-5219] Handle Azure instance record being split

* [COST-4745] OCPGCP Network data processing SQL (#5058)

* [COST-4745] OCPGCP Network data processing SQL

---------

Co-authored-by: Sam Doran <sdoran@redhat.com>

* [COST-5198] - split read traffic to read replica db using nginx proxy (#5188)

* update nginx with HTTP method routing
* switch koku-api to koku-api-writes
* duplicate koku-api to koku-api-reads add a optional mounted secret for the read replica
* update clowder configurator to read from read replica secret if mounted and enabled via ENV var

* remove unused methods (#5208)

* Bump certifi in the pip group across 1 directory (#5207)

* chore(image): update and rebuild image (#5203)

Co-authored-by: Update-a-Bot <insights@redhat.com>

* Handle case when resource ID cannot be obtained (#5209)

* Catch exception case.

* [COST-5148] filter out empty resource ids and SavingsPlanCoveredUsage entries (#5206)

* [COST-5148] update insert sql
filter out empty resource ids
offset savings from SavingsPlanCoveredUsage

* closing CASE statement

* clean up comment

* remove case stmts in favor of filtering out SavingsPlanCoveredUsage

* clean up

* Unpause the csi volume handle sql (#5175)

* update linting

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: cleaning up old changes. making first changes to create the API pieces.

* feat: insert new filters.

* feat: insert new filters and order by params.

* feat: customizing provider map and serializer.

* fix: changing TIME_CHOICES options.

* feat: first unit tests.

* feat: wip.

* feat: fixing provider map.

* feat: fixing provider map.

* update ec2 annotations to get all required fields

* use AWSEC2ComputeQueryParamSerializer

* feat: creating orderby and groupby serializers for ec2

* feat: unit tests for filters

* feat: removing group_by - not needed on ec2

* feat: fixing orderby serializer and starting units tests

* fix: typo

* feat: changing usage_hours to usage_amount

* feat: fixing unit test.

* wip: blocking some filters and unit test.

* feat: unit tests for group by filter

* flake8 fix

* feat: inserting more filters on validate function.

* feat: updating validate method to use similar logic and add filters.

* fix: changing unit tests for some filters.

* feat: testing filter combinations and flake8 checks.

* fix: test

* feat: serializer Unit tests and view Unit test fix

* flake8 fix

* fix: new approach to satisfy CodeCov

* fix:: getting rid of validate custom method.

* fix: commenting tags.

* fix: validarte functions, tests.

* handle filter params for specific report type

* transform tags to desired ui format

* default to monthly resolution on the EC2 endpoint

* add special pagination for EC2

* use default report type time period settings if exists

* fix typo

* fix: fixing parameters validations.

* update linting

* squash commits

* clean up query params and time period settings

* do not use filter keyword

* more code clean up
update unit tests

* address feedback
- move report_specific filters to main filter map
- use report_type instead of kwargs in get_paginator
- do not use deepcopy - just overwrite query_data
- resolution and time_scope_units are always monthly and month respectively
- overide start and end date params in base ParamSerializer
- overide limit and offset in base FilterSerializer

* more unit tests

* update openapi spec

* clean up and add unit tests

* move changes to openapi spec to a separate pr

* use serializer choice field and not customer validate method

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: David N <dnakabaa@redhat.com>
Co-authored-by: Luke Couzens <lcouzens@redhat.com>
Co-authored-by: Cody Myers <codymyers63@gmail.com>
Co-authored-by: Corey Goodfred <cgoodfre@redhat.com>
Co-authored-by: Sam Doran <sdoran@redhat.com>
Co-authored-by: Michael Skarbek <mskarbek@redhat.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Hambridge <chambrid@redhat.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Update-a-Bot <insights@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests smokes-required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants