Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test Case Recorder #87

Merged
merged 40 commits into from
Jul 31, 2021
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
f7587e3
Very preliminary steps
pokey Jul 3, 2021
815e829
Write test case info to file
brxck Jul 7, 2021
ed4ee0d
Save only target decorations
brxck Jul 7, 2021
3d0e8b0
Don't use yaml refs in test case fixtures
brxck Jul 8, 2021
e9b60e4
Record clipboard contents in snapshots
brxck Jul 9, 2021
efb696c
Run recorded test cases
brxck Jul 10, 2021
e95a993
Only save clipboard contents in copy/paste actions
brxck Jul 13, 2021
b06f27f
Reuse navigation map key creation
brxck Jul 13, 2021
c51def0
Add inferred full targets to fixture
brxck Jul 13, 2021
cc0e4bf
Move test fixtures from src to root
brxck Jul 14, 2021
9d65fde
Remove setting up navigation map
brxck Jul 14, 2021
fa52794
Attempt to mock clipboard (and fail)
brxck Jul 14, 2021
35a8ce5
Add navigationMap to graph
pokey Jul 13, 2021
0471c7f
Attempt to mock navigation map construction
brxck Jul 14, 2021
09a743f
Store "that mark" in test case fixture
brxck Jul 15, 2021
43a4473
Check nav map and improve that mark handling
brxck Jul 17, 2021
829882d
Give up for the night lol
brxck Jul 17, 2021
087c7fd
Support testing that mark
brxck Jul 18, 2021
50ff1e4
Add mockable clipboard layer
brxck Jul 18, 2021
f6db9e4
Test command return value
brxck Jul 18, 2021
be29b40
Only save targeted that marks to fixture
brxck Jul 18, 2021
84326b4
Name fixture and include Talon command
brxck Jul 18, 2021
c6cd5ec
Only save visible ranges for folding actions
brxck Jul 18, 2021
ca334b2
Start refactoring test cases
brxck Jul 27, 2021
01b8553
Fix that mark detection on compound targets
brxck Jul 27, 2021
c5c3d9b
Improve navigation map handling
brxck Jul 28, 2021
060d91d
Clarify thatMark exclusion conditions
brxck Jul 28, 2021
08ebc6b
Rename fixture fields
brxck Jul 28, 2021
076da28
Rephrase 'serialize' as 'to plain object'
brxck Jul 28, 2021
52f4408
Add issue for visible range testing
brxck Jul 28, 2021
ff6a1e1
Revert makeGraph export change
brxck Jul 28, 2021
f585de0
Improve fixture writing and move directory
brxck Jul 30, 2021
b8df923
Add test case folder selection
brxck Jul 30, 2021
decc22f
Create test case recorder class
brxck Jul 30, 2021
16f67bb
Ask for fixture directory after recording
brxck Jul 30, 2021
0b8df41
Move file handling to test case recorder
brxck Jul 30, 2021
3f188a3
Cleanup
pokey Jul 31, 2021
a255e0b
More cleanup
pokey Jul 31, 2021
8ce8f86
cleanup
pokey Jul 31, 2021
39d62f5
swap anchor with active
pokey Jul 31, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
10 changes: 9 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@
{
"command": "cursorless.recomputeDecorationStyles",
"title": "Cursorless: Recompute decoration styles"
},
{
"command": "cursorless.recordTestCase",
"title": "Cursorless: Record test case"
}
],
"colors": [
Expand Down Expand Up @@ -196,15 +200,19 @@
},
"devDependencies": {
"@types/glob": "^7.1.3",
"@types/js-yaml": "^4.0.2",
"@types/mocha": "^8.0.4",
"@types/node": "^12.11.7",
"@types/sinon": "^10.0.2",
"@types/vscode": "^1.53.0",
"@typescript-eslint/eslint-plugin": "^4.9.0",
"@typescript-eslint/parser": "^4.9.0",
"esbuild": "^0.11.12",
"eslint": "^7.15.0",
"glob": "^7.1.6",
"js-yaml": "^3.14.1",
"mocha": "^8.1.3",
"sinon": "^11.1.1",
"typescript": "^4.1.2",
"vscode-test": "^1.4.1"
},
Expand All @@ -213,4 +221,4 @@
"immutability-helper": "^3.1.1",
"lodash": "^4.17.21"
}
}
}
12 changes: 12 additions & 0 deletions src/Clipboard.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
import * as vscode from "vscode";

/**
* A mockable layer over the vscode clipboard
*
* For unknown reasons it's not possible to mock the clipboard directly.
* Use this instead of vscode.env.clipboard so it can be mocked in testing.
**/
export class Clipboard {
static readText = vscode.env.clipboard.readText;
static writeText = vscode.env.clipboard.writeText;
}
10 changes: 7 additions & 3 deletions src/NavigationMap.ts
Original file line number Diff line number Diff line change
Expand Up @@ -43,15 +43,19 @@ export default class NavigationMap {
[coloredSymbol: string]: Token;
} = {};

private getKey(color: SymbolColor, character: string) {
static getKey(color: SymbolColor, character: string) {
return `${color}.${character}`;
}

public addToken(color: SymbolColor, character: string, token: Token) {
this.map[this.getKey(color, character)] = token;
this.map[NavigationMap.getKey(color, character)] = token;
}

public getToken(color: SymbolColor, character: string) {
return this.map[this.getKey(color, character)];
return this.map[NavigationMap.getKey(color, character)];
}

public clear() {
this.map = {};
}
}
145 changes: 145 additions & 0 deletions src/TestCase.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
import * as path from "path";
import * as fs from "fs";
import * as yaml from "js-yaml";
import * as vscode from "vscode";
import NavigationMap from "./NavigationMap";
import { ThatMark } from "./ThatMark";
import { ActionType, PartialTarget, Target } from "./Types";
import { extractTargetedMarks } from "./extractTargetedMarks";
import { serializeMarks, SerializedMarks } from "./serializers";
import { takeSnapshot, TestCaseSnapshot } from "./takeSnapshot";

type TestCaseCommand = {
actionName: ActionType;
partialTargets: PartialTarget[];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For completeness I'd be tempted to also capture the full target, even if we don't end up using it for anything

extraArgs: any[];
};

type TestCaseContext = {
talonCommand: string;
thatMark: ThatMark;
targets: Target[];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a funny smell to have targets here but the actual cursorless command somewhere else

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow. Do you mean you would rather construct test case with a single argument/type ex. { command, talonCommand, thatMark, targets } ?

navigationMap: NavigationMap;
};

export type TestCaseFixture = {
talonCommand: string;
command: TestCaseCommand;
targets: Target[];
languageId: string;
marks: SerializedMarks;
initialState: TestCaseSnapshot;
finalState: TestCaseSnapshot;
returnValue: unknown;
};

export class TestCase {
talonCommand: string;
command: TestCaseCommand;
languageId: string;
targets: Target[];
marks: SerializedMarks;
context: TestCaseContext;
initialState: TestCaseSnapshot | null = null;
finalState: TestCaseSnapshot | null = null;
returnValue: unknown = null;

constructor(command: TestCaseCommand, context: TestCaseContext) {
const activeEditor = vscode.window.activeTextEditor!;
const { navigationMap, targets, talonCommand } = context;
const targetedMarks = extractTargetedMarks(targets, navigationMap);

this.talonCommand = talonCommand;
this.command = command;
this.languageId = activeEditor.document.languageId;
this.marks = serializeMarks(targetedMarks);
this.targets = targets;
this.context = context;
}

private includesThatMark(target: Target) {
if (target.type === "primitive" && target.mark.type === "that") {
return true;
} else if (target.type === "list") {
return target.elements.some(this.includesThatMark, this);
} else if (target.type === "range") {
return [target.start, target.end].some(this.includesThatMark, this);
}
return false;
}

private getExcludedFields() {
const excludableFields = {
clipboard: !["copy", "paste"].includes(this.command.actionName),
thatMark:
this.initialState == null &&
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not immediately obvious to me why we check whether initialState is null here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah looking at this I have to think about it too, maybe I can clarify the code.

If we are taking the first snapshot, then we only want to record thatMark when it is targeted. If we're taking the final snapshot, we always want to record thatMark.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I figured that out later; I'd just pass something into the function?

!this.targets.some(this.includesThatMark, this),
visibleRanges: ![
"fold",
"unfold",
"scrollToBottom",
"scrollToCenter",
"scrollToTop",
].includes(this.command.actionName),
};

return Object.keys(excludableFields).filter(
(field) => excludableFields[field]
);
}

private toYaml() {
if (this.initialState == null || this.finalState == null) {
throw Error("Two snapshots must be taken before serializing");
}
const fixture: TestCaseFixture = {
talonCommand: this.talonCommand,
languageId: this.languageId,
command: this.command,
targets: this.targets,
marks: this.marks,
initialState: this.initialState,
finalState: this.finalState,
returnValue: this.returnValue,
};
return yaml.dump(fixture, { noRefs: true, quotingType: '"' });
}

async recordInitialState() {
const excludeFields = this.getExcludedFields();
this.initialState = await takeSnapshot(
this.context.thatMark,
excludeFields
);
}

async recordFinalState(returnValue: unknown) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why unknown instead of any here? Tbh I don't really know the difference 😅

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's basically like a type safe any. You can use unknown in the same places you would use any, but it requires you to assert a type if you want to do anything with it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL thanks

const excludeFields = this.getExcludedFields();
this.returnValue = returnValue;
this.finalState = await takeSnapshot(this.context.thatMark, excludeFields);
}

async writeFile(filename: string) {
const fixture = this.toYaml();
const workspacePath = vscode.workspace.workspaceFolders?.[0].uri.path;
let document;

if (workspacePath && filename) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So are you assuming that the user is recording in the cursorless workspace?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, wasn't really sure how to handle this. I wasn't assuming we were in the cursorless workspace, and that's why I moved fixtures from src/test/suite/fixtures/recorded to a folder in the root.

Now that I'm thinking about it, I could actually check if we're in the cursorless workspace or not. If we're not in the cursorless workspace, just show the yaml in a pane?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that sounds good to me

Btw I'm happy to call this out of scope but from a usability standpoint I think my ideal sequence would be:

  1. Ask user for talon command
  2. Wait for them to run command
  3. Have subfolders in the fixture directory and present them with a list of possible subdirectories and allow them to select "add new" if they type something that doesn't exist
  4. Ask them for a file name
  5. Display a message that the fixture was made, with a link to open it, but don't open it by default

Note that steps 3–5 only happen if in cursorless workspace

So then what we'd do is to have a command in talon "record", which will grab the parsed command, call record on cursorless, type in literal command, then issue command. So workflow is then:

  1. Open a file to test on. We might consider storing a few common ones somewhere
  2. Say "record take air"
  3. Select subdirectory
  4. Say test name
  5. Reset file and repeat

Should be able to rapidly record dozens of tests the way. Alternately, we could keep things as they are extension side but ask for dir and name before asking for talon command. Then we'd have two record commands. The first one is eg "setup recording" and the second is "record". So you'd say

  1. "setup recording in named ", which would call extension record and fill out dir and name
  2. "record take air", which would type in "take air" literally, then issue the command

Maybe the second approach is better?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that sounds good to me

Btw I'm happy to call this out of scope but from a usability standpoint I think my ideal sequence would be:

  1. Ask user for talon command
  2. Wait for them to run command
  3. Have subfolders in the fixture directory and present them with a list of possible subdirectories and allow them to select "add new" if they type something that doesn't exist
  4. Ask them for a file name
  5. Display a message that the fixture was made, with a link to open it, but don't open it by default

Note that steps 3–5 only happen if in cursorless workspace

So then what we'd do is to have a command in talon "record", which will grab the parsed command, call record on cursorless, type in literal command, then issue command. So workflow is then:

  1. Open a file to test on. We might consider storing a few common ones somewhere
  2. Say "record take air"
  3. Select subdirectory
  4. Say test name
  5. Reset file and repeat

Should be able to rapidly record dozens of tests the way. Alternately, we could keep things as they are extension side but ask for dir and name before asking for talon command. Then we'd have two record commands. The first one is eg "setup recording" and the second is "record". So you'd say

  1. "setup recording in named ", which would call extension record and fill out dir and name
  2. "record take air", which would type in "take air" literally, then issue the command

Maybe the second approach is better?

const fixturePath = path.join(
workspacePath,
"testFixtures",
`${filename}.yml`
);
fs.writeFileSync(fixturePath, fixture);
document = await vscode.workspace.openTextDocument(fixturePath);
} else {
document = await vscode.workspace.openTextDocument({
language: "yaml",
content: fixture,
});
}
await vscode.window.showTextDocument(document, {
viewColumn: vscode.ViewColumn.Beside,
});
}
}
13 changes: 13 additions & 0 deletions src/ThatMark.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
import { SelectionWithEditor } from "./Types";

export class ThatMark {
private mark: SelectionWithEditor[] = [];

set(value: SelectionWithEditor[]) {
this.mark = value;
}

get() {
return this.mark;
}
}
1 change: 1 addition & 0 deletions src/Types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,7 @@ export type ActionRecord = Record<ActionType, Action>;
export interface Graph {
readonly actions: ActionRecord;
readonly editStyles: EditStyles;
readonly navigationMap: NavigationMap;
}

export interface DecorationColorSetting {
Expand Down
11 changes: 6 additions & 5 deletions src/addDecorationsToEditor.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,12 @@ interface CharacterTokenInfo {
tokenIdx: number;
}

export function addDecorationsToEditors(decorations: Decorations) {
export function addDecorationsToEditors(
navigationMap: NavigationMap,
decorations: Decorations
) {
navigationMap.clear();

var editors: vscode.TextEditor[];

if (vscode.window.activeTextEditor == null) {
Expand Down Expand Up @@ -85,8 +90,6 @@ export function addDecorationsToEditors(decorations: Decorations) {
])
);

const navigationMap = new NavigationMap();

// Picks the character with minimum color such that the next token that contains
// that character is as far away as possible.
// TODO: Could be improved by ignoring subsequent tokens that also contain
Expand Down Expand Up @@ -151,6 +154,4 @@ export function addDecorationsToEditors(decorations: Decorations) {
editor.setDecorations(decorations.decorationMap[color]!, ranges[color]!);
});
});

return navigationMap;
}