Support for Test at Scale Badge, run [TAS] #7612

nevilm-lt · 2022-02-14T13:39:18Z

Added

Added a badge support for Test At Scale

Usage

URL: http://localhost:8080/tas/tests/:PROVIDER/:ORGNAME/:REPONAME
Example Usage: http://localhost:8080/tas/tests/github/test-at-scale/demo-repo

Badge

shields-ci · 2022-02-14T13:40:49Z

	Messages
📖	✨ Thanks for your contribution to Shields, @nevilm-lt!

Generated by 🚫 dangerJS against 98448e6

calebcartwright · 2022-02-16T03:17:53Z

Thanks for the PR! Did one very quick glance over this and a few initial thoughts:

Please generalize the default label name to something like tests. That will ensure it's consistent with our other badges in the same category, and also ensure we aren't violating the promotional/branding aspects of our badge specification
We've got a number of utility functions for test-related badges that I'd encourage checking out (should make things easier!) For reference:
- https://github.com/badges/shields/blob/master/services/appveyor/appveyor-tests.service.js
- https://github.com/badges/shields/blob/master/services/sonar/sonar-tests.service.js

nevilm-lt · 2022-02-22T14:13:27Z

Hey @calebcartwright, Thanks for the suggestions.

As pointed out - I've updated the label to be more generic.
I've gone though the utility functions. But looks like our use case is more specific which can not be catered by functions already available. Please let me know if I'm missing something here I'll be happy to make those changes.

Thanks.

calebcartwright · 2022-02-22T16:22:13Z

Could you share some more info about the API surface? It gives me pause when I see "badge" as part of the path as it implies it's a specialized endpoint with minimal data for a single badge purpose. That's not necessarily a problem, but as I see it, part of the reason why you can't incorporate our existing functions (which provide end-user features around customization) is because the API response is so minimal.

I would think/hope there could be an endpoint that would provide some of the standard details around passed/failed/skipped/etc. that we see with the various other platforms we integrate with.

nevilm-lt · 2022-02-23T13:01:56Z

@calebcartwright On test-at-scale [TAS] we are not running all test cases for each build. We try to execute minimum number of test cases that are affected by the latest PR changes.
So the metric that we trying to display is the number of tests executed in "x" duration.

Here's the api example.
https://api.tas.lambdatest.com/repo/badge?git_provider=github&org=nevilm-lt&repo=cheerio
{"badge":{"status":"passed","duration":"27s","total_tests":"0"}}

calebcartwright · 2022-02-23T16:10:24Z

Thank you for the reply @nevilm-lt, but I'm not sure you've really responded to the point/topic.

We don't want Shields to be locked into an API from a platform that's trying to dictate a specific type of badge down to basic rendering concerns. We want to consume standard APIs that provide data points relevant to general users of the platform, such that we are then able to offer our users a range of options for badges as opposed to assuming/pushing a single use case.

We're absolutely on board with being able to provide our users with a TAS badge that displays the test count + duration, but I don't want us to be coupled to an API endpoint that only allows that badge because that really shouldn't be the only one.

nevilm-lt · 2022-02-24T11:50:30Z

Hi @calebcartwright,

We have updated the badge to show standard information as other badges provides. i.e. Passed, Failed, Skipped etc. Also we have used standard utility functions to render the badge. Please feel free to let us know if there are any other changes required.

Current Badge:

Thanks.

palashmarantas · 2022-02-25T06:51:25Z

Hi @calebcartwright,

We have updated the badge to show standard information as other badges provides. i.e. Passed, Failed, Skipped etc. Also we have used standard utility functions to render the badge. Please feel free to let us know if there are any other changes required.

Current Badge:

Thanks.

Any updates on this @calebcartwright ?

calebcartwright · 2022-02-25T23:57:34Z

Awesome, thanks for the updates! To clarify a bit on the prior discussion, if you'd like to be able to show a badge that includes the duration in addition to all the other variants that's perfectly alright as well.

Things like okay here based on a quick glance, so I can either give this a more thorough review as-is if you're fine with the existing badges, or I can hold off/suggest some next steps if you'd like to get a count+duration badge variant too.

palashmarantas · 2022-02-26T08:59:44Z

Awesome, thanks for the updates! To clarify a bit on the prior discussion, if you'd like to be able to show a badge that includes the duration in addition to all the other variants that's perfectly alright as well.

Things like okay here based on a quick glance, so I can either give this a more thorough review as-is if you're fine with the existing badges, or I can hold off/suggest some next steps if you'd like to get a count+duration badge variant too.

Hi @calebcartwright,
Thanks for response. Please provide thorough review for existing badge PR. Let's merge this PR first. Please share you points for count+duration will create separate PR for new badge

chris48s · 2022-02-27T19:54:40Z

services/tas/tas-tests.tester.js

+
+t.create('tas tests').get('/github/tasdemo/axios.json').expectBadge({
+  label: 'tests',
+  message: Joi.string(),


Can we make this more specific. This isn't really a useful test as an error message will match Joi.string() too. There are some validators in services/test-validators.js for testing this type of test summary badge:

shields/services/test-validators.js

Lines 135 to 154 in 6e80336

const isDefaultTestTotals = makeTestTotalsValidator({

passed: 'passed',

failed: 'failed',

skipped: 'skipped',

})

const isDefaultCompactTestTotals = makeCompactTestTotalsValidator({

passed: '✔',

failed: '✘',

skipped: '➟',

})

const isCustomTestTotals = makeTestTotalsValidator({

passed: 'good',

failed: 'bad',

skipped: 'n/a',

})

const isCustomCompactTestTotals = makeCompactTestTotalsValidator({

passed: '💃',

failed: '🤦‍♀️',

skipped: '🤷',

})

Also can we have a couple of tests for the behaviour in the 401 and 404 cases.

nevilm-lt · 2022-02-28T12:07:10Z

Hey @chris48s, We have added the test cases for multiple behaviours which mocks specific use cases. Please let us know if there are anything else we can do.

Thanks.

calebcartwright

Overall looks good to me! Believe we're in the home stretch now, couple final minor items

services/tas/tas-tests.tester.js

services/tas/tas-tests.service.js

calebcartwright · 2022-03-04T02:53:55Z

services/tas/tas-tests.service.js

+  namedParams: {
+    provider: 'github',
+    org: 'test-at-scale',
+    repo: 'badge-demo',
+  },


Sorry I missed this before but is this a valid, public project? I'm not seeing it on GitHub and the badge is unsurprisingly showing a not found. We really like to have these examples be valid projects so that users can actually see/work with them in the modal window.

If it's something you all are planning on setting up in the near future that's alright, but otherwise it would probably be better to swap in a real target

https://shields-staging-pr-7612.herokuapp.com/tas/tests/github/test-at-scale/badge-demo?compact_message

Using the same target as the tests would suffice too:

https://shields-staging-pr-7612.herokuapp.com/tas/tests/github/tasdemo/axios

@calebcartwright Thanks for pointing out. I've updated the example project.

calebcartwright

Awesome, thanks so much!

nevilm-lt and others added 2 commits February 8, 2022 23:00

added support for TAS application

4b276b6

Added prod url. Fixed test-case

c5fb71c

calebcartwright added the service-badge Accepted and actionable changes, features, and bugs label Feb 15, 2022

shields-cd temporarily deployed to shields-staging-pr-7612 February 17, 2022 01:28 Inactive

calebcartwright changed the title ~~Support for Test at Scale Badge~~ Support for Test at Scale Badge, run [TAS] Feb 17, 2022

nevilm-lt added 2 commits February 22, 2022 19:30

fixed label

cdfee73

updated tests

93e9dfe

shields-cd temporarily deployed to shields-staging-pr-7612 February 22, 2022 16:34 Inactive

streamlining of test badge

f56a2b5

shields-cd temporarily deployed to shields-staging-pr-7612 February 24, 2022 11:23 Inactive

chris48s reviewed Feb 27, 2022

View reviewed changes

mock test cases

070d737

shields-cd temporarily deployed to shields-staging-pr-7612 February 28, 2022 12:06 Inactive

calebcartwright reviewed Mar 1, 2022

View reviewed changes

services/tas/tas-tests.tester.js Outdated Show resolved Hide resolved

services/tas/tas-tests.service.js Outdated Show resolved Hide resolved

updated tests to make it live

8260f9e

shields-cd temporarily deployed to shields-staging-pr-7612 March 3, 2022 11:23 Inactive

updated schema validation

f0d2146

shields-cd temporarily deployed to shields-staging-pr-7612 March 3, 2022 11:30 Inactive

calebcartwright reviewed Mar 4, 2022

View reviewed changes

updated example

810236e

shields-cd temporarily deployed to shields-staging-pr-7612 March 4, 2022 05:41 Inactive

calebcartwright approved these changes Mar 5, 2022

View reviewed changes

calebcartwright added the squash when passing label Mar 5, 2022

Merge branch 'master' into feature-tas

39ee706

shields-cd temporarily deployed to shields-staging-pr-7612 March 5, 2022 01:27 Inactive

Merge branch 'master' into feature-tas

97f1e1d

shields-cd temporarily deployed to shields-staging-pr-7612 March 5, 2022 01:29 Inactive

Merge branch 'master' into feature-tas

aee1f14

shields-cd temporarily deployed to shields-staging-pr-7612 March 5, 2022 01:38 Inactive

Merge branch 'master' into feature-tas

4342757

shields-cd temporarily deployed to shields-staging-pr-7612 March 5, 2022 01:45 Inactive

Merge branch 'master' into feature-tas

dcccf30

shields-cd temporarily deployed to shields-staging-pr-7612 March 5, 2022 01:53 Inactive

Merge branch 'master' into feature-tas

98448e6

shields-cd deployed to shields-staging-pr-7612 March 5, 2022 02:00 View deployment

repo-ranger bot merged commit 5ae9fd7 into badges:master Mar 5, 2022

PyvesB mentioned this pull request Feb 10, 2024

deprecate TAS #9932

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for Test at Scale Badge, run [TAS] #7612

Support for Test at Scale Badge, run [TAS] #7612

nevilm-lt commented Feb 14, 2022 •

edited

shields-ci commented Feb 14, 2022 •

edited

calebcartwright commented Feb 16, 2022

nevilm-lt commented Feb 22, 2022

calebcartwright commented Feb 22, 2022

nevilm-lt commented Feb 23, 2022

calebcartwright commented Feb 23, 2022

nevilm-lt commented Feb 24, 2022 •

edited

palashmarantas commented Feb 25, 2022

calebcartwright commented Feb 25, 2022

palashmarantas commented Feb 26, 2022 •

edited

chris48s Feb 27, 2022

nevilm-lt Mar 3, 2022

nevilm-lt commented Feb 28, 2022

calebcartwright left a comment

calebcartwright Mar 4, 2022

nevilm-lt Mar 4, 2022

calebcartwright left a comment

	const isDefaultTestTotals = makeTestTotalsValidator({
	passed: 'passed',
	failed: 'failed',
	skipped: 'skipped',
	})
	const isDefaultCompactTestTotals = makeCompactTestTotalsValidator({
	passed: '✔',
	failed: '✘',
	skipped: '➟',
	})
	const isCustomTestTotals = makeTestTotalsValidator({
	passed: 'good',
	failed: 'bad',
	skipped: 'n/a',
	})
	const isCustomCompactTestTotals = makeCompactTestTotalsValidator({
	passed: '💃',
	failed: '🤦‍♀️',
	skipped: '🤷',
	})

Support for Test at Scale Badge, run [TAS] #7612

Support for Test at Scale Badge, run [TAS] #7612

Conversation

nevilm-lt commented Feb 14, 2022 • edited

Added

Usage

Badge

shields-ci commented Feb 14, 2022 • edited

calebcartwright commented Feb 16, 2022

nevilm-lt commented Feb 22, 2022

calebcartwright commented Feb 22, 2022

nevilm-lt commented Feb 23, 2022

calebcartwright commented Feb 23, 2022

nevilm-lt commented Feb 24, 2022 • edited

palashmarantas commented Feb 25, 2022

calebcartwright commented Feb 25, 2022

palashmarantas commented Feb 26, 2022 • edited

chris48s Feb 27, 2022

Choose a reason for hiding this comment

nevilm-lt Mar 3, 2022

Choose a reason for hiding this comment

nevilm-lt commented Feb 28, 2022

calebcartwright left a comment

Choose a reason for hiding this comment

calebcartwright Mar 4, 2022

Choose a reason for hiding this comment

nevilm-lt Mar 4, 2022

Choose a reason for hiding this comment

calebcartwright left a comment

Choose a reason for hiding this comment

nevilm-lt commented Feb 14, 2022 •

edited

shields-ci commented Feb 14, 2022 •

edited

nevilm-lt commented Feb 24, 2022 •

edited

palashmarantas commented Feb 26, 2022 •

edited