Merge branch 'main' into CO-731-A-proof-of-concept-that-sends-the-AI-…

…Element-image-to-the-inference-with-inference-changes

CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## [0.18.1](https://github.com/askui/askui/compare/v0.18.0...v0.18.1) (2024-06-13)
+
+
+### Features
+
+* **askui-nodejs:** add new custom element options (#CL-731) ([2e9d294](https://github.com/askui/askui/commit/2e9d294fd9182a97465cf32d080f348a5a121af4)), closes [#CL-731](https://github.com/askui/askui/issues/CL-731)
+* **ML-492:** use default api version as v4 ([7898fb7](https://github.com/askui/askui/commit/7898fb7758d8b94f76e757163b200df4cc5c31db))
+* **ML-492:** use V4 online learning endpoint. ([d6c2923](https://github.com/askui/askui/commit/d6c2923f143a7642de6c601a446327bff92dbdea))
+
+
+### Bug Fixes
+
+* **askui-nodejs:** fix retry of http requests, e.g., don't retry on user mistakes (#CL-736) ([73781eb](https://github.com/askui/askui/commit/73781eb46facd2af1519150627b9ca77c3ced215)), closes [#CL-736](https://github.com/askui/askui/issues/CL-736)
+* **askui-nodejs:** log (not throw) error if reporter is misused to reveal underlying error (#CL-736) ([1c8de05](https://github.com/askui/askui/commit/1c8de05f0bd6495b32e3a8c3f8044a53b0a36681)), closes [#CL-736](https://github.com/askui/askui/issues/CL-736)
+* **ML-492:** change url to experimental ([e8e40e0](https://github.com/askui/askui/commit/e8e40e078cd034fafd9bae7dce5432386f1bf8e7))
+* **ML-492:** fix url to consistent ([e3915d8](https://github.com/askui/askui/commit/e3915d8e4ac41ef63ad3f7e573745f3fe5af1b6d))
+* **ML-492:** fix v4 isImageRequired logic ([ee47c11](https://github.com/askui/askui/commit/ee47c1100577fc22c58e6584e3df26a3947d5cc9))
+* **ML-492:** linting fix for alphabetical ([46e7055](https://github.com/askui/askui/commit/46e7055bed7210edcd7adf9215a6333dd179786e))
+* **ML-492:** linting fix for alphabetical ([0f2c7e0](https://github.com/askui/askui/commit/0f2c7e018e9c2d4b160368f1a1e478592d33fa6d))
+* **ML-492:** linting fixes ([223cf94](https://github.com/askui/askui/commit/223cf940ecb9452248990ca26cce6f178cd17aaf))
+
+
 ## [0.17.1](https://github.com/askui/askui/compare/v0.17.0...v0.17.1) (2024-04-15)
 
 

packages/askui-nodejs/package.json

@@ -1,6 +1,6 @@
 {
     "name": "askui",
-    "version": "0.18.0",
+    "version": "0.18.1",
     "license": "MIT",
     "author": "askui GmbH <info@askui.com> (http://www.askui.com/)",
     "description": "Reliable, automated end-to-end-testing that depends on what is shown on your screen instead of the technology you are running on",

packages/askui-nodejs/src/core/model/custom-element-json.ts

@@ -1,17 +1,16 @@
 /**
  * Defines a 'custom element'. This is a UI element which is defined by
  * providing an image and other parameters such as degree of rotation.
- * It allows filtering for a UI element that is not recognized
- * by our machine learning models by default.
- * It can also be used for pixel assertions of elements using classical
- * [template matching](https://en.wikipedia.org/wiki/Template_matching).
+ * It allows filtering for a UI element based on an image instead of using
+ * text or element descriptions like `button().withText('Submit')` in
+ * `await aui.click().button().withText('Submit').exec()`.
  *
- * **Important:** The `CustomElementJson` needs to capture as accurately as possible
- * what the custom element looks like during test execution as otherwise
- * our machine learning models cannot find it, even with the additional data
- * provided. This is especially true for the resolution used while cropping
- * the `CustomElementJson.customImage` which should match the resolution during
- * test execution.
+ * **Important:** The `CustomElementJson` needs to capture as accurately as
+ * possible what the custom element looks like during test execution as
+ * otherwise our machine learning models cannot find it, even with the
+ * additional data provided. This is especially true for the resolution used
+ * while cropping the `CustomElementJson.customImage` which should be as
+ * similar as possible to the resolution during test execution.
  *
  * Rotated custom elements can be filtered for using
  * `CustomElementJson.rotationDegreePerStep`.
@@ -48,9 +47,24 @@ export interface CustomElementJson {
    * probably not what you want) and `1.0` (= elements need to look exactly
    * like defined by `CustomElementJson` which is unlikely to be achieved
    * as even minor differences count). Defaults to `0.9`.
+   *
+   * **Important**: The `threshold` impacts the prediction quality.
    */
   threshold?: number | undefined,
 
+  /**
+   * A threshold for when to stop searching for UI elements similar to the
+   * custom element. As soon as UI elements have been found that are at least
+   * as similar as the `stopThreshold`, the search is going to stop. After that
+   * elements are filtered using the `threshold`. Because of that the
+   * `stopThreshold` should be greater than or equal to `threshold`. It is
+   * primarily to be used as a speed improvement (by lowering the value).
+   * Takes values between `0.0` and `1.0`. Defaults to `0.9`.
+   *
+   * **Important**: The `stopThreshold` impacts the prediction speed.
+   */
+  stopThreshold?: number | undefined,
+
   /**
    * A step size in rotation degree. Rotates the custom image by
    * `rotationDegreePerStep` until 360° is exceeded. Range is between
@@ -64,13 +78,17 @@ export interface CustomElementJson {
   /**
    * A color compare style. Allows matching a custom element by color, e.g.,
    * instead of filtering for all icons (blue and green ones),
-   * only the green ones captured by `customImage` are filtered for using 'RGB'.
-   * Defaults to 'grayscale'.
+   * only the green ones captured by `customImage` are filtered for using
+   * 'RGB'. Defaults to 'grayscale'.
    *
-   * **Important**: This increases the prediction time quite a bit. So only use
-   * it when absolutely necessary.
+   * **Important**: The `imageCompareFormat` impacts the prediction time as
+   * well as quality. Although this highly depends on the use case, the other
+   * parameters and the actual scene in which to find the UI element, as a
+   * rule of thumb, 'edges' is likely to be a bit faster than 'grayscale' and
+   * 'grayscale' is likely to be a bit faster than 'RGB'. For quality it is
+   * most often the other way around.
    */
-  imageCompareFormat?: 'RGB' | 'grayscale' | undefined,
+  imageCompareFormat?: 'RGB' | 'grayscale' | 'edges' | undefined,
 
   /** A polygon to match only a certain area of the custom element. */
   mask?: ({ x: number; y: number; })[] | undefined,

packages/askui-nodejs/src/core/model/custom-element.spec.ts

@@ -9,6 +9,7 @@ describe('CustomElement', () => {
         base64ImageString,
         'Dummy_element',
         0.7,
+        0.8,
         10,
         'RGB',
         [{ x: 0, y: 1 }, { x: 1, y: 2 }, { x: 3, y: 4 }],
@@ -19,6 +20,7 @@ describe('CustomElement', () => {
         mask: [{ x: 0, y: 1 }, { x: 1, y: 2 }, { x: 3, y: 4 }],
         name: 'Dummy_element',
         rotationDegreePerStep: 10,
+        stopThreshold: 0.8,
         threshold: 0.7,
       });
       expect(actual).toStrictEqual(expected);
@@ -30,6 +32,7 @@ describe('CustomElement', () => {
           base64ImageString,
           'Dummy_element',
           1.1,
+          0.8,
           10,
           'RGB',
           [{ x: 0, y: 1 }, { x: 1, y: 2 }, { x: 3, y: 4 }],
@@ -38,12 +41,28 @@ describe('CustomElement', () => {
       }).toThrow('threshold must be less than or equal to 1');
     });
 
+    test('should throw ValidationError if stopThreshold is invalid', () => {
+      expect(() => {
+        const customElement = new CustomElement(
+          base64ImageString,
+          'Dummy_element',
+          0.7,
+          1.1,
+          10,
+          'RGB',
+          [{ x: 0, y: 1 }, { x: 1, y: 2 }, { x: 3, y: 4 }],
+        );
+        customElement.validate();
+      }).toThrow('stopThreshold must be less than or equal to 1');
+    });
+
     test('should throw ValidationError if rotationDegreePerStep is invalid', () => {
       expect(() => {
         const customElement = new CustomElement(
           base64ImageString,
           'Dummy_element',
           0.9,
+          0.9,
           -90,
           'RGB',
           [{ x: 0, y: 1 }, { x: 1, y: 2 }, { x: 3, y: 4 }],
@@ -58,6 +77,7 @@ describe('CustomElement', () => {
           base64ImageString,
           'Dummy_element',
           0.9,
+          0.9,
           10,
           'RGB',
           [{ x: 0, y: 1 }, { x: 1, y: 2 }],
@@ -72,6 +92,7 @@ describe('CustomElement', () => {
           base64ImageString,
           'Dummy_element',
           90,
+          0.8,
           10,
           'RGB',
           [{ x: 0, y: 1 }, { x: 1, y: 2 }],

packages/askui-nodejs/src/core/model/custom-element.ts

@@ -8,15 +8,17 @@ export class CustomElement implements CustomElementJson {
   private static schema = object({
     mask: array().optional().min(3, 'mask must contain at least 3 points'),
     rotationDegreePerStep: number().optional().min(0).lessThan(360),
+    stopThreshold: number().optional().min(0).max(1),
     threshold: number().optional().min(0).max(1),
   });
 
   constructor(
     public customImage: string,
     public name?: string,
     public threshold?: number,
+    public stopThreshold?: number,
     public rotationDegreePerStep?: number,
-    public imageCompareFormat?: 'RGB' | 'grayscale',
+    public imageCompareFormat?: 'RGB' | 'grayscale' | 'edges',
     public mask?: ({ x: number; y: number; })[],
   ) {
   }
@@ -46,6 +48,7 @@ export class CustomElement implements CustomElementJson {
       ceJson.customImage,
       ceJson.name,
       ceJson.threshold,
+      ceJson.stopThreshold,
       ceJson.rotationDegreePerStep,
       ceJson.imageCompareFormat,
       ceJson.mask,

packages/askui-nodejs/src/core/reporting/step-reporter.ts

@@ -85,40 +85,49 @@ export class StepReporter {
 
   async onStepBegin(snapshot: Snapshot): Promise<void> {
     if (this._currentStep === undefined) {
-      throw new Error('Cannot begin step if step is undefined.');
+      logger.error('Cannot begin step if step is undefined.');
+      return Promise.resolve();
     }
 
     if (this._currentStep.status !== 'pending') {
-      throw new Error('Cannot begin step that is not pending.');
+      logger.error('Cannot begin step that is not pending.');
+      return Promise.resolve();
     }
 
     this._currentStep = this._currentStep.onBegin(snapshot);
     this.enqueueReporterCalls('onStepBegin', this._currentStep);
+    return Promise.resolve();
   }
 
   async onStepRetry(snapshot: Snapshot, error: Error): Promise<void> {
     if (this._currentStep === undefined) {
-      throw new Error('Cannot retry step if step is undefined.');
+      logger.error('Cannot retry step if step is undefined.');
+      return Promise.resolve();
     }
 
     if (this._currentStep.status !== 'running') {
-      throw new Error('Cannot retry step that has not been running.');
+      logger.error('Cannot retry step that has not been running.');
+      return Promise.resolve();
     }
 
     this._currentStep = this._currentStep.onRetry(snapshot, error);
     this.enqueueReporterCalls('onStepRetry', this._currentStep);
+    return Promise.resolve();
   }
 
   async onStepEnd(snapshot: Snapshot, error?: Error): Promise<void> {
     if (this._currentStep === undefined) {
-      throw new Error('Cannot end step if step is undefined.');
+      logger.error('Cannot end step if step is undefined.');
+      return Promise.resolve();
     }
 
     if (this._currentStep.status !== 'running') {
-      throw new Error('Cannot end step that has not been running.');
+      logger.error('Cannot end step that has not been running.');
+      return Promise.resolve();
     }
 
     this._currentStep = this._currentStep.onEnd(snapshot, error);
     this.enqueueReporterCalls('onStepEnd', this._currentStep);
+    return Promise.resolve();
   }
 }

packages/askui-nodejs/src/execution/dsl.ts

@@ -517,7 +517,8 @@ export class FluentFilters extends FluentBase {
    *
    * // optional parameter: similarity_score
    * '978-0-201-00650-6' == withText('978-0-201-00') => true with 82.76 similarity
-   * '978-0-201-00650-6' == withText('978-0-201-00650', 90) => true with 93.75 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => false with 82.76 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => true with 93.75 < 90 similarity
    * ```
    * ![](https://docs.askui.com/img/gif/withText.gif)
    *
@@ -647,13 +648,9 @@ export class FluentFilters extends FluentBase {
    * The text description inside the `matching()` should describe the element visually.
    * It understands color, some famous company/product names, general descriptions.
    *
-   * **Important: _Matching only returns the best matching element when you use it with `get()`_**
-   *
-   * A bit of playing around to find a matching description is sometimes needed:
-   * E.g., `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
-   * Generally, the more detail the better.
-   *
-   * We also recommend to not restrict the type of element by using the generalselector `element()` as shown in the examples below.
+   * It sometimes requires a bit of playing around to find a matching description:
+   * E.g. `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
+   * Generally the more detail the better.
    *
    * **Examples:**
    * ```typescript
@@ -1472,7 +1469,8 @@ export class FluentFiltersCondition extends FluentBase {
    *
    * // optional parameter: similarity_score
    * '978-0-201-00650-6' == withText('978-0-201-00') => true with 82.76 similarity
-   * '978-0-201-00650-6' == withText('978-0-201-00650', 90) => true with 93.75 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => false with 82.76 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => true with 93.75 < 90 similarity
    * ```
    * ![](https://docs.askui.com/img/gif/withText.gif)
    *
@@ -1602,13 +1600,9 @@ export class FluentFiltersCondition extends FluentBase {
    * The text description inside the `matching()` should describe the element visually.
    * It understands color, some famous company/product names, general descriptions.
    *
-   * **Important: _Matching only returns the best matching element when you use it with `get()`_**
-   *
-   * A bit of playing around to find a matching description is sometimes needed:
-   * E.g., `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
-   * Generally, the more detail the better.
-   *
-   * We also recommend to not restrict the type of element by using the generalselector `element()` as shown in the examples below.
+   * It sometimes requires a bit of playing around to find a matching description:
+   * E.g. `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
+   * Generally the more detail the better.
    *
    * **Examples:**
    * ```typescript
@@ -3242,7 +3236,8 @@ export class FluentFiltersGetter extends FluentBase {
    *
    * // optional parameter: similarity_score
    * '978-0-201-00650-6' == withText('978-0-201-00') => true with 82.76 similarity
-   * '978-0-201-00650-6' == withText('978-0-201-00650', 90) => true with 93.75 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => false with 82.76 < 90 similarity
+   * '978-0-201-00650-6' == withText('978-0-201-00', 90) => true with 93.75 < 90 similarity
    * ```
    * ![](https://docs.askui.com/img/gif/withText.gif)
    *
@@ -3372,13 +3367,9 @@ export class FluentFiltersGetter extends FluentBase {
    * The text description inside the `matching()` should describe the element visually.
    * It understands color, some famous company/product names, general descriptions.
    *
-   * **Important: _Matching only returns the best matching element when you use it with `get()`_**
-   *
-   * A bit of playing around to find a matching description is sometimes needed:
-   * E.g., `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
-   * Generally, the more detail the better.
-   *
-   * We also recommend to not restrict the type of element by using the generalselector `element()` as shown in the examples below.
+   * It sometimes requires a bit of playing around to find a matching description:
+   * E.g. `puzzle piece` can fail while `an icon showing a puzzle piece` might work.
+   * Generally the more detail the better.
    *
    * **Examples:**
    * ```typescript
@@ -3809,9 +3800,6 @@ export abstract class Getter extends FluentCommand {
    *
    * **Examples:**
    * ```typescript
-   * // ************************************ //
-   * // Log the DetectedElements completely  //
-   * // ************************************ //
    * const text = await aui.get().text('Sign').exec();
    * console.log(text);
    *

packages/askui-nodejs/src/execution/index.ts

@@ -1 +1 @@
-export { UiControlClient, RelationsForConvenienceMethods } from './ui-control-client';
+export * from './ui-control-client';