Skip to content

Commit d504e38

Browse files
isc-sveluryisc-svelury
committed
First attempt at a Test Plan for v2.0.0 and a framework for manual testing
1 parent 03b8710 commit d504e38

File tree

2 files changed

+204
-0
lines changed

2 files changed

+204
-0
lines changed

‎Manual Testing Framework.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Manual Testing Framework
2+
3+
## Testing Flow
4+
5+
1. Setup and configuration
6+
2. Backend and IRIS level
7+
3. Frontend
8+
9+
## Creating the test plan
10+
11+
1. List all changes since the last release by going through the commits on `main`.
12+
2. Identify the exact methods/parts that need to be tested.
13+
3. For each part that needs to be tested:
14+
1. **Objective and Groundwork**:
15+
- How should it be tested: The objective, whether explicitly defined or not, should inform how we want to test a piece of work.
16+
- Degree of thoroughness: Is this part critical and/or has many test cases etc.
17+
2. **Initial Setup**: Work that needs to be done before working on any and all test cases.
18+
3. **Test cases**: For each test case
19+
1. Setup (if necessary)
20+
2. Input (for Frontend testing, we need to test both VS Code and Studio)
21+
3. Expected output
22+
4. Document results (NOT A PART OF INITIAL TEST PLAN)
23+
5. Cleanup (if necessary)
24+
25+
## Documenting Results
26+
27+
Document anything unexpected or abnormal throughout the whole process
28+
29+
1. Include steps to reproduce the issue
30+
2. (Very Optional) Potential reasons you might have thought of.
31+
3. (Even More Optional) Potential fixes you might have thought of.

‎Pre-release Test Plan.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# Pre-Release Manual Test Plan
2+
3+
## Summary of Changes (since last release)
4+
5+
### Features
6+
7+
1. Uncommitted check on checkout:
8+
- On branch checkout, the WebUI requests information about uncommitted changes from `api/uncommitted` and confirms that the user knows another user has uncommitted changes and that they will carry them over on checkout.
9+
- This applies to checking out existing local and remote branches and creating new local branches.
10+
2. Single file delete: If a package was in source control, then it was not possible to remove a single file or group of files from source control.
11+
- Original behaviour:
12+
- Now, when a package is added to source control we go through the whole package and add its individual files to source control. When this feature was developed, the package and its sub-packages were also added to source control.
13+
- When the user tries to remove a single file, if it is source control, it is removed.
14+
- If the user tries to remove a package, then everything pertaining to that package is removed.
15+
- The issue with this approach comes from the fact that there isn't an easy way to determine whether a package (`.pkg` file) should be in source control. For adding and removing to work as expected with the right options being displayed, it required a lot of book-keeping. This leads to edge cases that may or may not be covered, resulting in unexpected, and potentially blocking, behaviour.
16+
- Behaviour after [PR #125](https://github.com/intersystems/git-source-control/pull/125):
17+
- For packages, both options — Add and Remove are displayed. We are no longer concerned with figuring out whether a package should be in source control - it never is!
18+
- No packages or sub-packages should be added to source control. When the user adds a package to source control, all its files not already in source control get added.
19+
- When the user removes a package from source control, all its files in source control will get removed.
20+
- We don't have to worry about any weird edge cases this way and behaviour is more consistent overall.
21+
3. Navigation between Settings page and WebUI:
22+
- We did not support URL prefixes initially and added that in later. It would be worthwhile to check if there is something else along that vein that is missing. The same session should be used for both pages as well.
23+
24+
### Bug Fixes
25+
26+
1. Line break to output window: In `SourceControl.Git.Change.CLS` Craig added a new line to the output.
27+
2. Wrap mapping creation in else block: This was a small bug fix for the flat folder structure for certain file types enhancement to the mappings.
28+
3. Changed the name of the % class replace field to be correct.
29+
4. Added newlines to outputs wherever necessary for better formatting.
30+
31+
### Documentation
32+
33+
1. Help modal update
34+
35+
### Technical Debt
36+
37+
1. Refactor WebUI Code:
38+
- A majority of the changes were related to the sidebar.
39+
- All the event handlers were moved, and the accordion was also modified.
40+
- I also collapsed the handlers for local and remote checkout and local and remote branch merge into a checkout handler and merge handler respectively. So those should be tested to make sure there aren't any issues.
41+
42+
### Enhancements
43+
44+
1. Configurability for how the "%" symbol is changed on export of `%` classes.
45+
2. Removed `sc-list.txt` from the project. Everything should be working smoothly without it. Testing Add, Remove and Pull would be good.
46+
47+
<br>
48+
49+
50+
<br>
51+
52+
## Stuff to test
53+
54+
### Setup and Configuration
55+
56+
1. `%` class export configurability.
57+
2. `NoFolders` toggle (coming soon!)
58+
59+
### Backend/IRIS
60+
61+
1. **Single file removal from source control**: We need to test this make sure we are removing files correctly from the filesystem, IRIS and source control without affecting anything that shouldn't be.
62+
2. **No more `sc-list.txt`**: We need to make sure that `^SYS("SourceControl", "Git","items")` gets updated correctly every time the there is a change to list of tracked files &mdash; Add, Remove, Merge. One failure case I know is if the user does a fetch and then does a merge with a remote branch from the WebUI they will have technically executed a git pull that is independent of the package and any changes to the list of tracked files. I am not sure of the implications of the global being out of sync with git.
63+
64+
### Frontend
65+
1. **Uncommitted check on checkout**: We need to check that files edited by another user are in fact being passed in the response from `api/uncommitted` and make sure that the prompt is being displayed as needed.
66+
2. **Navigation between Settings page and WebUI**: We need to test that the URL prefix (if any) prepends the URL correctly, and that the CSP session is shared. Initially we did not support URL prefixes. I don't know if there is something else in there that I might have missed.
67+
3. **Sidebar operations**: We need to double-check all the sidebar operations after the refactoring to make sure those are all still good and all the prompts and stuff make sense.
68+
69+
### Things to watch for
70+
1. Check if the output on the terminal has appropriate spacing.
71+
2. Check the text on the settings page input names and tooltips.
72+
3. Check the text in the prompts in the WebUI
73+
4. Double-check the text in the help modal.
74+
75+
<br>
76+
77+
78+
<br>
79+
80+
## The Actual Tests
81+
82+
### Setup
83+
84+
Testing environment:
85+
86+
1. Create a local repository
87+
2. Create a remote repository
88+
3. Configure as usual
89+
90+
### Setup and Configuration Related Tests
91+
92+
1. **`%` class configurability**:
93+
1. Setup:
94+
1. Add "_" as the `%` replacement character.
95+
2. Test cases:
96+
1. Add a `%` class to source control
97+
2. Remove the class from source control.
98+
3. Expected Results:
99+
1. Class should be exported as "_<classname_without_percent>".
100+
2. This should remove the file from source control and delete the external file.
101+
102+
2. **`NoFolders` toggle**
103+
3. **Both together**
104+
105+
### Backend/IRIS
106+
107+
1. **Single file removal from source control**: We need to test this make sure we are removing files correctly from the filesystem, IRIS and source control without affecting anything that shouldn't be.
108+
1. Setup:
109+
1. Create a package with 2-3 sub-packages and classes and CSP pages distributed in them.
110+
2. Create a project with the new package in it.
111+
3. Add the whole project into source control.
112+
2. Test Cases:
113+
1. Remove one file from source control
114+
2. Remove a group of files (not adding up to a whole package) from source control
115+
3. Remove a `.csp` file from source control
116+
4. Remove a group of `.csp` files from source control
117+
5. Remove a package from source control &mdash; test for packages and sub-packages.
118+
6. Remove a project from source control
119+
3. Expected Results:
120+
1. For all test cases, the corresponding file(s) should be deleted externally. Their entries in `^SYS("SourceControl", "Git","items")` should be deleted correctly. They should no longer be tracked by git.
121+
4. Notes:
122+
1. Removing CSP files and projects from source control has not been tested ever.
123+
2. **No more `sc-list.txt`**: We need to make sure that `^SYS("SourceControl", "Git","items")` gets updated correctly every time the there is a change to list of tracked files &mdash; Add, Remove, Merge. One failure case I know is if the user does a fetch and then does a merge with a remote branch from the WebUI they will have technically executed a git pull that is independent of the package and any changes to the list of tracked files. I am not sure of the implications of the global being out of sync with git.
124+
1. Test Cases:
125+
1. Add a file to source control from VS Code or Studio: This will be covered in the setup portion of Single file removal from source control.
126+
2. Add a file to source control from management portal.
127+
3. Add a file to the remote and perform a git pull.
128+
2. Expected Results:
129+
1. Verify that the global is updated, and the file is exported and being tracked by git.
130+
2. Verify that the global is updated, and the file is exported and being tracked by git.
131+
3. Verify that the global is updated, and the file is imported.
132+
133+
### Frontend
134+
1. **Uncommitted check on checkout**: We need to check that files edited by another user are in fact being passed in the response from `api/uncommitted` and make sure that the prompt is being displayed as needed.
135+
1. Setup:
136+
1. Create 2 files &mdash; one that also lives in IRIS (`TestClass.cls`) and one that doesn't (`TestFile.md`) with a different IRIS user than the one being used for testing.
137+
2. Test Cases (for each file): There are 3 types of changes, 3 types of checkouts and 2 types of files to consider here giving us 18 possible scenarios. We also need to test all these on both VS Code and Studio. (36 test cases in all)
138+
1. Types of changes:
139+
1. File creation by another user
140+
2. File edits by another user
141+
3. File delete by another user
142+
2. Types of checkouts:
143+
1. Local branch checkout
144+
2. Local branch creation
145+
3. Remote branch checkout
146+
3. Types of files:
147+
1. Tracked by git and IRIS (`TestClass.cls`)
148+
2. Tracked only by git (`TestFile.md`)
149+
3. Expected Results:
150+
1. In all 9 cases, changes to `TestFile.md`, should not be marked as changes by a different user.
151+
2. Changes to `TestClass.cls` should be marked as changes by a different user and a prompt should come up warning you that you will be carrying over changes by another user if you proceed with the checkout.
152+
3. The actual checkout might not work. You might see something like "changes will be overwritten" in the error modal that pops up. This is okay.
153+
2. **Navigation between the Settings page and WebUI**: We need to test that prefixes are being appended correctly and sessions are being shared. Initially we did not support URL prefixes. I don't know if there is something else like that.
154+
1. Setup:
155+
1. Add a URL prefix and open either the settings page or the WebUI.
156+
2. Test Cases (VS Code and Studio):
157+
1. From the Settings page, go to the WebUI. Check that the URL prefix is preserved and the CSP session is shared.
158+
2. Go back to the WebUI and again confirm that the URL prefix is present, and we are still in the same CSP session.
159+
3. Expected Results:
160+
1. Everything should just work. We shouldn't see any error with respect to navigation in both Studio and VS Code.
161+
3. **Sidebar operations**: We need to double-check all the sidebar operations after the refactoring to make sure those are all still good and all the prompts and stuff make sense. I have been using them often, so I don't think there should be any issues, but best to be safe.
162+
1. Test Cases (VS Code and Studio): There are 4 local branch operations and 3 remote branch operations.
163+
1. Local Branch Operations:
164+
1. Branch creation: covered in Uncommitted check on checkout
165+
2. Existing branch checkout: covered in Uncommitted check on checkout
166+
3. Merge branches
167+
4. Delete branch
168+
2. Remote Branch Operations:
169+
1. Prune remote branches
170+
2. Existing branch checkout: covered in Uncommitted check on checkout
171+
3. Merge branches
172+
2. Expected Results:
173+
1. All the operations should be executed correctly. Git might throw errors and warnings, but that's okay.

0 commit comments

Comments
 (0)