Added test plan for TypeSpec extension

[xiaohua899]

fix #5444
Description

Added four test scenarios test plan to TypeSpec Extension:

• create-typespec-project-test-plan.md
• generate-from-typespec-test-plan.md
• import-typespec-from-openapi3-test-plan.md
• typespec-extension-basic-features-test-plan.md

@RodgeFu and @msyyc for notification.

[xiaohua899]

@microsoft-github-policy-service agree company="Microsoft"

[timotheeguerin]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

[RodgeFu]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

@msyyc , could you help to double check the images to see whether all of them are necessary for vender to follow to do the test? Timothee has a point that images have a higher maintain cost, so it's good if we can simplify the images but still make sure the doc is good enough for venders to follow to test the features. thanks.

After these user experience in UX is stabilized, we will expect any changes in these UX in the future should contains the corresponding change in these manual test plan (just like updating auto test for feature update now) and go through a manual test sign-off by venders which should also be helpful to make sure the doc is up-to-date. thanks.

[msyyc]

I think some of the concern is that's a lot of image for a git repo that might need to be updated as the test plan change

@msyyc , could you help to double check the images to see whether all of them are necessary for vender to follow to do the test? Timothee has a point that images have a higher maintain cost, so it's good if we can simplify the images but still make sure the doc is good enough for venders to follow to test the features. thanks.

After these user experience in UX is stabilized, we will expect any changes in these UX in the future should contains the corresponding change in these manual test plan (just like updating auto test for feature update now) and go through a manual test sign-off by venders which should also be helpful to make sure the doc is up-to-date. thanks.

Got it.
(1) We will try if we could reduce the number of images to least. For example, maybe we could reuse some images.
(2) In last sync meeting, I already told vendors to make an investigation about auto test based on the test doc. And as you said, we should make sure to update the test doc once there is new feature or change.

[timotheeguerin]

Is it also not possible to automate this instead

[RodgeFu]

Is it also not possible to automate this instead

we are doing some investigation into the auto-test for it as @msyyc mentioned. It will depend on the investigation result and how frequently these UX will be changed later to determine whether we want to add auto UI test (I have dealt with UI test a lot when in Sql which is actually painful to maintain because of its instability in nature). Before that, manual test would help to cover these user experience in vscode. thanks.

[markcowl]

I think the major concern is that this repo is the core repo for typespec and a test document does not seem to belong. If we need to check a test plan into someplace, I would suggest some other repo (perhaps typespec-azure-pr).

This just does not seem like it is of general interest to consumers of typespec, so does not seem to belong in the repo

[msyyc]

Update: after optimization, we reduce the image number used by docs from 80 to 55.

[RodgeFu]

I think the major concern is that this repo is the core repo for typespec and a test document does not seem to belong. If we need to check a test plan into someplace, I would suggest some other repo (perhaps typespec-azure-pr).

This just does not seem like it is of general interest to consumers of typespec, so does not seem to belong in the repo

I think we should call them "manual test cases" instead of "test plan" which would be more accurate. They are actually the same as automated test in logic if we don't look at the implementation (code/doc) and execution engine (machine/people). And a few detail reasons why I would prefer them to be along with the code:

• It's easier for people to find the corresponding test to "run" when they are changing the code. For example, if some people (internal or external) want to contribute to the UX part code in vscode, they can run these "manual test" to make sure nothing is broken when submitting their change. (just like they need to make sure all the automated tests pass when contributing into other features)
• It's also easier to make sure these test cases are always up to date with the code. For example, later on, when people make change impacting these UX experience in vscode, they need also update these tests(doc) in the same PR properly so that vender can use the updated test case(doc) in the PR to "run" the test and sign-off. Otherwise, these tests will fail if vender verifying using the original test case. (just like people needs to update automated test accordingly when they make some change impacting tests, otherwise the test execution will fail)

thx

[xiaohua899]

@RodgeFu /@msyyc Regarding the trigger of Import TypeSpec from OpenAPI3, I changed it to three options in the test plan, please review.

[xiaohua899]

@RodgeFu and @chunyu3 , Updated the test plan document based on #6032 . Please review.

@markcowl The PR has been updated according to your suggestion.

1. Change to use "test case" instead of "test plan".
2. Add a contributing.md.
3. Remove anything Microsoft-specific.

CC: @msyyc

[timotheeguerin]

Talking with the team

1. this is still a lot of images which when I look at it provide very little value but add size to this repo, they just show basic vscode UI and folder structure which could just be very well be described with a few words and be easier to maintain
2. manual test are still a nightmare to deal with and put a lof of extra work on us as well
3. There is already quite a few prs in progress that are changing things

[RodgeFu]

actually we are making some progress in the ui-automation and we may go to that approach directly. A quick question, does our CI machine have UI? or are we able to have some machine with UI in CI so that we will be able to run these UI test in CI? thanks.

[timotheeguerin]

Can't the ui run headless, windows is super slow in ci and Linux doesn't.
None of the other ui test we have have any issues

[RodgeFu]

Where are these ui test? We would like to have a look into them

[timotheeguerin]

there is 2(3 if you include the azure repo)

• packages/playground-website test:e2e script that runs playwright
• packages/typespec-vscode itself that runs the basic browser smoke test

[RodgeFu]

cool, thx! @msyyc , could you help to double check these tests and see whether there is anything our ui-test can leverage? thanks.

[msyyc]

Copy that.

[timotheeguerin]

https://github.com/vitest-dev/vscode/tree/main/test-e2e vitest vscode extension also has a bunch of ui test, also running with playwright+ vitest