Automate BM usage calculations

[joachimweyl]

Motivation

We will have more BM users as time goes by. We also want to be able to charge for lease time only.

Completion Criteria

CSV with BM usage by project with cost associated.

Description

• Create 3 new tables to pull from
  • Project Table - filled with info we need to gather when we create new ESI projects
    • ESI project - value from ESI project column
    • Invoice Project name - Coldfront Project name
    • PI - we will have to ask when we create the ESI project or if owner column is filled we can use that
    • Start Time
    • End Time
    • Project type
      • Tracked cluster - Grafana query is how we track them (see Grafana querry below)
      • Non-tracked cluster - Lease time is how we invoice them
      • Other - Lease time is how we invoice them
  • Server Types - or maybe this should be added to nerc-rates?
    • Server
      • BM FC430 - fc430 in ESI Resource Column
      • BM FC830 - fc830 in ESI Resource Column
      • BM GPUA100SXM4 - lenovo-sd650nv2-a100 or sd650nv2 in ESI Resource Column
      • BM GPUH100 - no ESI Resource Column yet but once we have it we want to add it here
    • Cost - per hour
      • $0.75
      • $3.97
      • $8.312
      • $24.16
  • BM Usage
    • Invoice Month
    • Project - Allocation
    • Project - Allocation ID
    • Manager (PI)
    • Invoice Email
    • Invoice Address
    • Institution
    • Institution - Specific Code
    • SU Hours (GBhr or SUhr)
    • SU Type
    • Rate
    • Cost
    • Credit
    • Credit Code
    • Balance
• Gather data from ESI at 12:01 am on the 1st of the month
  • OpenStack esi lease list --status any --long should give you a list to pull from
    • this will have data for each lease with a start, end and expire time, we need to SUM all lease time during the previous month for each node for each project.
      • If Start time is earlier than first of month use first of month
      • If end time is later than expire time use expire time otherwise use end time
        • If above is later than end of month use end of month
  • With that information, we can fill in BM Usage CSV when combined with the other tables we created
  • Many of the BM Usage columns will be empty but the ones below will be filled
  • Invoice Month is the previous month in this format "2024-10"
  • Project - Allocation
    • for example, ESI project "Noodles" becomes "Noodels BM Usage" links from
  • Project - Allocation ID becomes "ESI Bare Metal"
  • Manager (PI) comes from the Project Table above
  • Invoice Email is "nclinton@bu.edu" for now
  • Institution comes from the Project Table above
  • SU Hours (GBhr or SUhr) comes from the summation we did above
  • Rate comes from Server Types or nerc-rates whichever we decided to hold this info
  • Cost Rate * SU Hours
  • Balance Cost-Credit (which is zero so basically just cost again)
• Setup a list of non-billable projects
  • Current list
    • admin
    • ajamias
    • anshul
    • cloudlab
    • d4n
    • danni
    • Edge-Manager
    • flightctl-dev
    • FlinkEnergy
    • hakan
    • hwbroken
    • hypershift-cluster
    • ibm_research
    • jappavoo
    • larsks
    • MOCPerf
    • naved
    • nerc-admins
    • nerc-test
    • okd
    • okdci
    • okdci_hosts
    • ope
    • orran_cloud_computing
    • racedoro
    • sina_ahmadi
    • skanthed
    • storage-optimization
    • storage-research
    • Triton
    • tzumainn
    • uitest
    • ukl
• Test that output looks similar to this to the right of column Invoice Month

Grafana Query

sum by (cluster) (
  sum_over_time(
    (max_over_time(DCGM_FI_DEV_GPU_UTIL[1h:]) > bool 0)
  [$__range:1h])
)

Completion dates

Desired - 2025-01-26
Required - TBD

[joachimweyl]

@QuanMPhm This is worth starting to think about.

[QuanMPhm]

@knikolla @naved001 From this issue, if you guys are fine, I will write-up the following sub-issues:

• In nerc-rates, to include BM SU rates
• In non-billlable-projects, to include the new list of nonbillable projects
• In invoicing, to implement new invoice class for BM usage

@joachimweyl I have a few invoicing questions:

• Will BM projects be included in all the current invoices as normal? (I.e in billable, NERC-total, etc)
• I assume some of these SUs will need to be in the Lenovo invoice?
• Is BM eligible for any of our current credits, subsidies, or prepayments?
• In the list of nonbillable projects you provided above, do they only apply if for BM usage? (I.e if the jappavoo project uses both BM and regular CPU or GPU SUs, is only the BM usage nonbillable, or both the BM and regular usage?)

@knikolla @naved001 I assume more work will need to be done in openshift-usage-scripts and openstack-billing-from-db once we iron out details on how to obtain and store BM usage data.

[joachimweyl]

1. Yes BM projects will be included in all current invoices as normal
2. Correct A100SXM4s and eventually whatever we call the H100s
3. No BM is not eligible for any credits, that being said they might eventually have their own credits so we should make sure there is a way to enable disable credits for different types of "clusters" in this case we would call this "cluster" BM, but eventually, we might have different credits for ocp-test vs ocp-prod vs some other cluster, does it make sense to track this in a table? something like rates repo?
4. For now we could probably get away with sharing the same nonbillable table as there should be no overlap with the names in OpenShift/OpenStack vs BM. That being said the same as credits I think we will need to eventually filter differently per cluster so we might want to start thinking about this now.

[joachimweyl]

@QuanMPhm are you still using this issue for this work or are you breaking this out into multiple issues? I am happy to help break this out and then close this one so that we no longer have a huge 8 story point item.

[QuanMPhm]

@joachimweyl I am breaking this into smaller issues written in other repos.

[joachimweyl]

Once all the other issues exist feel free to remove story points from this one and close it. That way we don't have this large item sitting around.

[QuanMPhm]

@joachimweyl I have created all the smaller issues in repos elsewhere. I suppose it is fine to close this issue now?

[hpdempsey]

I don't think we had a design for this did we? Not thrilled about proceeding with implementation without that.

…

On Thu, Feb 27, 2025 at 10:00 AM Quan Pham ***@***.***> wrote: @joachimweyl <https://github.com/joachimweyl> I have created all the smaller issues in repos elsewhere. I suppose it is fine to close this issue now? — Reply to this email directly, view it on GitHub <#831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMGMIELOI5T3QDKTVNJ2R2D2R4SBBAVCNFSM6AAAAABSJ56XLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBYGIYTMMZUGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***> [image: QuanMPhm]*QuanMPhm* left a comment (nerc-project/operations#831) <#831 (comment)> @joachimweyl <https://github.com/joachimweyl> I have created all the smaller issues in repos elsewhere. I suppose it is fine to close this issue now? — Reply to this email directly, view it on GitHub <#831 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMGMIELOI5T3QDKTVNJ2R2D2R4SBBAVCNFSM6AAAAABSJ56XLCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBYGIYTMMZUGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[joachimweyl]

@hpdempsey This issue is automating a process that I have been doing manually. John, Wayne, and Scott already agreed to the format when it was created manually. That being said, I understand the desire for a requirements document. I will update the Invoicing Requirements document with the BM addition.

[joachimweyl]

Updates done.

[QuanMPhm]

Closing this as it has now been divided into issues elsewhere:
CCI-MOC/nerc-rates#14
CCI-MOC/invoicing#152
https://github.com/CCI-MOC/non-billable-projects/pull/44
CCI-MOC/bare-metal-usage-scripts#1