Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stub out API test cases related to Reports #95

Closed
kdelee opened this issue Dec 8, 2017 · 4 comments
Closed

Stub out API test cases related to Reports #95

kdelee opened this issue Dec 8, 2017 · 4 comments
Assignees
Milestone

Comments

@kdelee
Copy link
Contributor

kdelee commented Dec 8, 2017

This is another area where we need to stub out tests for functionality that does not exist much yet, so they need to be relatively high level.

Tests for all reports:

  • Test that any fact included in any report may be obfuscated/redacted/forward encrypted upon "export" (whatever that may be)
  • Test that I may “tag” hosts (ideally via DNS name or network IP address) with “exclude” attribute to communicate to SEAP that these hosts should be considered as paid for through some other arrangement (Example: AWS instance paid through royalty by Amazon itself)

Test by Report type:

  • "Raw Report" -- host by host breakdown of all facts collected, rho-like
    • general correctness test like rho
    • unlike rho, it seems we want some determination of what is there (not just contents of /etc/redhat-release which is a string, but a boolean or ternary fact that says YES/NO/MAYBE this is RHEL 4)
  • "Density Reports"
    • Test that for each product, I can generate a “density report” that includes the following data about about hosts found with product including:
      • # hosts
      • # of virtual cores
      • # of physical cores
      • # of sockets
      • # of socket pairs
      • Version of product ?
      • Other detail about how we ascertained that the product was present?
      • Report includes sum of virtual cores
      • Report includes sum of physical cores
      • Report includes sum of sockets
      • Report includes sum of socket pairs
      • Report correctly differentiates between redhat and community versions of product
      • Report deduces from raw facts definitive version of rhel and when it cannot deduce definitive version, it indicates this in some way (unknown version?)
      • Report indicates what fact collections this report is sourced from
  • "Time series" reports
    • Test that for each product (will we regard EAP 6.y and 7.y the same product?), I can generate a that takes in many fact collections given a start date and end date and source and produces a report that shows how many {hosts, physical cores, sockets, socket pairs, virtual cores} were found with that product at each scan. Must allow some definition of “step value”, i.e. period between fact collections to include.

One outstanding question regarding Density and Raw reports if they will also aggregate data from multiple scans and do any deduplication. Time series by definition must take in multiple fact collections, but these would obstesibly be from different points in time. It is unclear if they must be able to merge fact collections from the same "time step"

@kdelee kdelee added this to the Sonar 1.0 Test Stubs milestone Dec 8, 2017
@kdelee
Copy link
Contributor Author

kdelee commented Dec 8, 2017

We should suggest that time series reports get punted to the future, because this does not make sense without scheduling

@kdelee
Copy link
Contributor Author

kdelee commented Dec 8, 2017

Sample "density report" as I imagine it:

JBOSS EAP DENISTY REPORT    
hosts cores sockets
one.com 27.5 1
two.com 6 2
three.com 8 3
     
summary 41.5 6

Obviously the 27.5 cores is silly

@kdelee
Copy link
Contributor Author

kdelee commented Jan 3, 2018

@kdelee kdelee modified the milestones: Sprint 13, Sprint 14 Jan 4, 2018
@kdelee kdelee self-assigned this Jan 23, 2018
@kdelee
Copy link
Contributor Author

kdelee commented Jan 25, 2018

Also add test stub (functionality not implemented yet) for redacting fields of reports at some "export" stage

kdelee added a commit that referenced this issue Jan 25, 2018
Outline test cases for reports. Reports are still very much in flux and
we need some more details about what facts will be present, with what
priority they are chosen between sources when different information is
merged, and what queries to the reports endpoint will be acceptable.
That being said, this is a good start and gets the test cases in a
format so we can view them from our read the docs website.

Closes #95
Closes #94
Closes #98
kdelee added a commit that referenced this issue Jan 25, 2018
Outline test cases for reports. Reports are still very much in flux and
we need some more details about what facts will be present, with what
priority they are chosen between sources when different information is
merged, and what queries to the reports endpoint will be acceptable.
That being said, this is a good start and gets the test cases in a
format so we can view them from our read the docs website.

Closes #95
Closes #94
Closes #98
kdelee added a commit that referenced this issue Jan 26, 2018
Outline test cases for reports. Reports are still very much in flux and
we need some more details about what facts will be present, with what
priority they are chosen between sources when different information is
merged, and what queries to the reports endpoint will be acceptable.
That being said, this is a good start and gets the test cases in a
format so we can view them from our read the docs website.

Closes #95
Closes #94
Closes #98
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant