support element snapshot (#18)

integrate testing with appium desktop and fixed the logical coordinates issue

README.md

@@ -6,7 +6,7 @@ This repo is an open source asp.net core implementation of WinAppDriver and it's
 
 I name this project YWinAppDriver(yet another WinAppDriver).
 
-## Project Status: 0.2.2
+## Project Status: 0.2.x
    Most of functionalities(except Pen, Touch and Mouse) are ready and you should be able to switch from WinAppDriver to YWinAppDriver without any(or With little) change. [SessionController.cs](https://github.com/licanhua/YWinAppDriver/blob/main/src/WinAppDriver/Controllers/SessionController.cs) defines all the endpoints.
 
    I successfully made [CalculatorTest](https://github.com/licanhua/YWinAppDriver/tree/main/examples/CalculatorTest) and [WebDriverAPI](https://github.com/licanhua/YWinAppDriver/tree/main/test/WebDriverAPI) work which comes from [WinAppDriver samples](https://github.com/microsoft/WinAppDriver/tree/master/Samples/C%23/CalculatorTest) and [WinAppDriver test](https://github.com/microsoft/WinAppDriver/tree/master/Tests/WebDriverAPI)
@@ -17,12 +17,6 @@ I name this project YWinAppDriver(yet another WinAppDriver).
   - logs. Complete
   - XPath. Complete. For the XPath syntax, refer to https://docs.microsoft.com/en-us/previous-versions/dotnet/netframework-4.0/ms256086(v=vs.100)
 
-## Not ready features
-Below features are not ready yet
-- Mouse and Touch Input code completed, but I didn't fully test it. WinUI functionality in [InputHelper.cs](https://github.com/microsoft/microsoft-ui-xaml/blob/master/test/testinfra/MUXTestInfra/Common/InputHelper.cs)
-- Integration testing with appium
-- Extension the WinAppDriver functionality by ExecuteScript
-
 ## Download & Run YWinAppDriver
 - Download and compile
 
@@ -32,6 +26,7 @@ There are two ways to get the WinAppDriver.exe:
 
 - Lauch WinAppDriver.exe
 Generally speaking, WinAppDriver user would have two settings: http://127.0.0.1:4723 or http://127.0.0.1:4723/wd/hub.
+Note: [dotnet-core runtime](https://dotnet.microsoft.com/download/dotnet-core/3.1) is required, please install 3.1 or late it if you have problem to run WinAppDriver.exe
 
 By default, YWinAppDriver is http://127.0.0.1:4723. You can change the port number and basepath easily:
 1. CLI
@@ -116,7 +111,8 @@ Because YWinAppDriver is for desktop application other than browser, `no` below
 |maybe|POST	|/session/:sessionId/refresh	|Refresh the current page.
 |maybe|POST	|/session/:sessionId/execute	|Inject a snippet of JavaScript into the page for execution in the context of the currently selected frame.
 |maybe|POST|	/session/:sessionId/execute_async	|Inject a snippet of JavaScript into the page for execution in the context of the currently selected frame.
-|complete|GET|	/session/:sessionId/screenshot	|Take a screenshot of the current page.(Implementation: Full screen snapshot only)
+|complete|GET|	/session/:sessionId/screenshot	|Take a screenshot of the current page
+|complete|GET|	/session/:sessionId/element/:elementId/screenshot	|Take a screenshot of the element
 |no|GET|	/session/:sessionId/ime/available_engines	|List all available engines on the machine.
 |no|GET|	/session/:sessionId/ime/active_engine	|Get the name of the active IME engine.
 |no|GET|	/session/:sessionId/ime/activated	|Indicates whether IME input is active at the moment (not if it's available.
@@ -170,14 +166,14 @@ Because YWinAppDriver is for desktop application other than browser, `no` below
 |completed|POST|	/session/:sessionId/buttondown	|Click and hold the left mouse button (at the coordinates set by the last moveto command).
 |completed|POST|	/session/:sessionId/buttonup	|Releases the mouse button previously held (where the mouse is currently at).
 |completed|POST|	/session/:sessionId/doubleclick	|Double-clicks at the current mouse coordinates (set by moveto).
-|not tested|POST|	/session/:sessionId/touch/click	|Single tap on the touch enabled device.
-|not tested|POST|	/session/:sessionId/touch/down	|Finger down on the screen.
-|not tested|POST|	/session/:sessionId/touch/up	|Finger up on the screen.
-|not tested|POST|	session/:sessionId/touch/move	|Finger move on the screen.
+|completed|POST|	/session/:sessionId/touch/click	|Single tap on the touch enabled device.
+|completed|POST|	/session/:sessionId/touch/down	|Finger down on the screen.
+|completed|POST|	/session/:sessionId/touch/up	|Finger up on the screen.
+|completed|POST|	session/:sessionId/touch/move	|Finger move on the screen.
 |in progress|POST|	session/:sessionId/touch/scroll	|Scroll on the touch screen using finger based motion events.
 |in progress|POST|	session/:sessionId/touch/scroll	|Scroll on the touch screen using finger based motion events.
-|not testeds|POST|	session/:sessionId/touch/doubleclick	|Double tap on the touch screen using finger motion events.
-|not tested|POST|	session/:sessionId/touch/longclick	|Long press on the touch screen using finger motion events.
+|completed|POST|	session/:sessionId/touch/doubleclick	|Double tap on the touch screen using finger motion events.
+|completed|POST|	session/:sessionId/touch/longclick	|Long press on the touch screen using finger motion events.
 |in progress|POST|	session/:sessionId/touch/flick	|Flick on the touch screen using finger motion events.
 |in progress|POST|	session/:sessionId/touch/flick	|Flick on the touch screen using finger motion events.
 |no|GET	|/session/:sessionId/location	|Get the current geo location.
@@ -242,12 +238,17 @@ Below are the capabilities that can be used to create Windows Application Driver
 | attachToTopLevelWindowClassName  	| app should be "Root", Existing application top level window to attach to. if you are using WinAppDriver, please use appTopLevelWindow    	| `0xB822E2`                                            	|
 | appWorkingDir      	| Application working directory (Classic apps only)     	| `C:\Temp`                                             	|
 | forceMatchAppTitle	| If app is launched, but have problem to match it, YWinAppDriver do the last try to match with the application title	| Calculator                                               	|
-| forceMatchClassName	| If app is launched, but have problem to match it, YWinAppDriver do the last try to match with the class name	| Calculator                                               	|
+| forceMatchClassName	| If app is launched, but have problem to match it, YWinAppDriver do the last try to match with the class name	| Chrome_WidgetWin_1                                               	|
 
-## Known issue
+## YWinAppDriver addressed some WinAppDriver 1.2 issues and fixed them
 
-### Window is not moved exactly the position you want.
+### WinAppDriver has problem to start c:\windows\system32\calc.exe, PostMan, or Notepad++ [issue 1372](https://github.com/microsoft/WinAppDriver/issues/1372)
+In YWinAppDriver, you can workaround the problem with the capabilites like below.
+```
+"forceMatchAppTitle": "Calculator"
+"forceMatchClassName":"Notepad++"
+```
 
-The window is moved, but it didn't pass the API testing. I guess the Window position is not the same as the control position, so there is some adjustment before the move. But I didn't find this API yet.'
-/session/:sessionId/window/:windowHandle/position
-/session/:sessionId/window/:windowHandle/size
+### Appium desktop can't show all controls WinAppDriver provided
+ [Appium Desktop](https://github.com/appium/appium-desktop) is a great tool to inspect the app's elements and it's very easy to learn and use it.
+ Currently WinAppDriver can't show all elements. I don't know if it's the issue of WinAppDriver or Appium, but YWinAppDriver doesn't have this problem.

src/Infra/CommandHandler/SessionHandlers.cs

@@ -198,9 +198,18 @@ protected override object ExecuteSessionCommand(ISessionManager sessionManager,
 
   class TakeScreenshotHandler : SessionCommandHandlerBase<string>
   {
-    protected override string ExecuteSessionCommand(ISessionManager sessionManager, ISession session, string ignored)
+    protected override string ExecuteSessionCommand(ISessionManager sessionManager, ISession session, string elementId)
     {
-      return session.TakeScreenshot();
+      // bring the root window to top first
+      var windowHandle = session.GetApplicationRoot().UI().NativeWindowHandle;
+      Helper.Helpers.SetForegroundWindow(windowHandle);
+      Helper.Helpers.BringWindowToTop(windowHandle);
+
+      var element = elementId == null ? session.GetApplicationRoot() : session.FindElement(elementId);
+      var desktopWindow = element.GetDesktopRectangle();
+      var elementWindow = element.GetBoundingRectangle();
+      elementWindow.IntersectsWith(desktopWindow);
+      return session.TakeScreenshot(elementWindow.X, elementWindow.Y, elementWindow.Height, elementWindow.Width);
     }
   }
 

src/Infra/Communication/DummyElement.cs

@@ -1,5 +1,6 @@
 using System;
 using System.Collections.Generic;
+using System.Drawing;
 using System.Text;
 using System.Xml;
 using WinAppDriver.Infra.Result;
@@ -152,5 +153,15 @@ public void SendKeys(string keys)
     {
       throw new NoSuchWindow();
     }
+
+    public Rectangle GetBoundingRectangle()
+    {
+      throw new NoSuchWindow();
+    }
+
+    public Rectangle GetDesktopRectangle()
+    {
+      throw new NoSuchWindow();
+    }
   }
 }

src/Infra/Communication/Element.cs

@@ -9,6 +9,7 @@
 using Microsoft.Windows.Apps.Test.Foundation.Waiters;
 using System;
 using System.Collections.Generic;
+using System.Drawing;
 using System.Linq;
 using System.Text;
 using System.Text.RegularExpressions;
@@ -26,7 +27,7 @@ public class Element : IElement
     const string UNIQID = "UniqId";
 
     private readonly UIObject _uiObject;
-    private readonly string id;
+    private string _id;
     private void EnsureTimeoutSettting(int msTimeout)
     {
       if (msTimeout == 0)
@@ -41,16 +42,18 @@ private void EnsureTimeoutSettting(int msTimeout)
       }
     }
 
-    public Element(UIObject uIObject, bool uniqId = false)
+    public void GenerateGuidAsId()
+    {
+      _id = Guid.NewGuid().ToString();
+    }
+    public Element(UIObject uIObject)
     {
       _uiObject = uIObject;
-      if (_uiObject != null && !uniqId)
-      {
-        id = _uiObject.RuntimeId;
-      }
-      if (id == null)
+
+      _id = _uiObject.RuntimeId;
+      if (_id == null)
       {
-        id = Guid.NewGuid().ToString();
+        GenerateGuidAsId();
       }
     }
 
@@ -157,7 +160,7 @@ public IEnumerable<IElement> FindElements(Locator locator, int msTimeout)
 
     public string GetId()
     {
-      return id;
+      return _id;
     }
 
     public bool IsSelected()
@@ -236,11 +239,18 @@ public string GetText()
       return _uiObject.Name;
     }
 
+    // Add an indirect node to make appiumdesktop v1.18.3 happy
+    private void BuildXmlDoc(XmlDocument doc, XmlNode node)
+    {
+      var body = doc.CreateElement("Body");
+      body.AppendChild(node);
+      doc.AppendChild(body);
+    }
     public XmlDocument GetXmlDocument()
     {
       var doc = new XmlDocument();
       var node = BuildXmlNode(doc, _uiObject, null);
-      doc.AppendChild(node);
+      BuildXmlDoc(doc, node);
       return doc;
     }
 
@@ -250,7 +260,7 @@ public IElement FindElementByXPath(string xpath)
 
       var doc = new XmlDocument();
       var xmlNode = BuildXmlNode(doc, _uiObject, uniqIdCache);
-      doc.AppendChild(xmlNode);
+      BuildXmlDoc(doc, xmlNode);
 
       var node = doc.SelectSingleNode(xpath);
       if (node == null)
@@ -271,7 +281,7 @@ public IEnumerable<IElement> FindElementsByXPath(string xpath)
 
       var doc = new XmlDocument();
       var xmlNode = BuildXmlNode(doc, _uiObject, uniqIdCache);
-      doc.AppendChild(xmlNode);
+      BuildXmlDoc(doc, xmlNode);
 
       var nodes = doc.SelectNodes(xpath);
       foreach (XmlNode node in nodes)
@@ -302,15 +312,24 @@ private XmlNode BuildXmlNode(XmlDocument doc, UIObject obj, Dictionary<string, I
     {
       var element = doc.CreateElement(GetXamlNodeTag(obj));
       element.SetAttribute("AutomationId", System.Net.WebUtility.HtmlEncode(obj.AutomationId));
-      element.SetAttribute("Location", System.Net.WebUtility.HtmlEncode(obj.BoundingRectangle.ToString()));
+      element.SetAttribute("x", System.Net.WebUtility.HtmlEncode(obj.BoundingRectangle.X.ToString()));
+      element.SetAttribute("y", System.Net.WebUtility.HtmlEncode(obj.BoundingRectangle.Y.ToString()));
+      element.SetAttribute("width", System.Net.WebUtility.HtmlEncode(obj.BoundingRectangle.Width.ToString()));
+      element.SetAttribute("height", System.Net.WebUtility.HtmlEncode(obj.BoundingRectangle.Height.ToString()));
+      element.SetAttribute("IsOffScreen", System.Net.WebUtility.HtmlEncode(obj.IsOffscreen.ToString()));
+      element.SetAttribute("IsEnabled", System.Net.WebUtility.HtmlEncode(obj.IsEnabled.ToString()));
       element.SetAttribute("ClassName", System.Net.WebUtility.HtmlEncode(obj.ClassName));
       element.SetAttribute("LocalizedControlType", System.Net.WebUtility.HtmlEncode(obj.LocalizedControlType));
       element.SetAttribute("Name", System.Net.WebUtility.HtmlEncode(obj.Name));
       element.SetAttribute("RuntimeId", System.Net.WebUtility.HtmlEncode(obj.RuntimeId));
 
       if (uniqIdCache != null)
       {
-        var e = new Element(obj, true);
+        var e = new Element(obj);
+        if (uniqIdCache.ContainsKey(e.GetId()))
+        {
+          e.GenerateGuidAsId();
+        }
         element.SetAttribute(UNIQID, e.GetId());
         uniqIdCache[e.GetId()] = e;
       }
@@ -324,7 +343,7 @@ private XmlNode BuildXmlNode(XmlDocument doc, UIObject obj, Dictionary<string, I
 
     public string GetAttribute(LocatorStrategy locator)
     {
-      if (locator == LocatorStrategy.Id) return id;
+      if (locator == LocatorStrategy.Id) return _id;
       else if (locator == LocatorStrategy.AccessibilityId) return _uiObject.AutomationId;
       else if (locator == LocatorStrategy.ClassName) return _uiObject.ClassName;
       else if (locator == LocatorStrategy.TagName) return _uiObject.LocalizedControlType;
@@ -390,7 +409,7 @@ private UIObject GetActiveWindow()
     }
     public IElement GetWindowHandle()
     {
-      return new Element(GetActiveWindow(), true);
+      return new Element(GetActiveWindow());
     }
 
     private IEnumerable<UIObject> GetWindows()
@@ -412,7 +431,7 @@ private IEnumerable<UIObject> GetWindows()
 
     public IEnumerable<IElement> GetWindowHandles()
     {
-      return GetWindows().Select(window => new Element(window, true));
+      return GetWindows().Select(window => new Element(window));
     }
 
     public void SetFocus()
@@ -476,7 +495,7 @@ public string GetAttribute(string attributeName)
       // support https://github.com/microsoft/WinAppDriver/blob/master/Tests/WebDriverAPI/ElementAttribute.cs
       if (attributeName == ActionStrings.RuntimeId)
       {
-        return id;
+        return _id;
       }
       else if (attributeName == "name")
       {
@@ -502,5 +521,15 @@ public IEnumerable<IElement> GetChildren()
     {
       return _uiObject.Children.Select(item => new Element(item));
     }
+
+    public Rectangle GetBoundingRectangle()
+    {
+      return _uiObject.BoundingRectangle;
+    }
+
+    public Rectangle GetDesktopRectangle()
+    {
+      return UIObject.Root.BoundingRectangle;
+    }
   }
 }

src/Infra/Communication/IElement.cs

@@ -2,6 +2,7 @@
 // Licensed under the MIT License.
 
 using System.Collections.Generic;
+using System.Drawing;
 using System.Text;
 using System.Xml;
 using WinAppDriver.Infra.Result;
@@ -40,5 +41,9 @@ public interface IElement
     bool ElementEquals(IElement element);
     object GetUIObject();
     public IElement GetFocusedElement();
+    // UI Element screen size
+    Rectangle GetBoundingRectangle();
+    // Windows desktop screen size
+    Rectangle GetDesktopRectangle();
   }
 }

src/Infra/Helper/Helpers.cs

@@ -0,0 +1,22 @@
+using System;
+using System.Collections.Generic;
+using System.Runtime.InteropServices;
+using System.Text;
+
+namespace WinAppDriver.Infra.Helper
+{
+  public class Helpers
+  {
+    public const int PROCESS_PER_MONITOR_DPI_AWARE = 0x000000002;
+
+    [DllImport("shcore.dll", CharSet = CharSet.Unicode)]
+    public static extern uint SetProcessDpiAwareness(int dpiAwareness);
+
+    [DllImport("user32.dll")]
+    [return: MarshalAs(UnmanagedType.Bool)]
+    public static extern bool SetForegroundWindow(IntPtr hWnd);
+
+    [DllImport("user32.dll", SetLastError = true)]
+    public static extern bool BringWindowToTop(IntPtr hWnd);
+  }
+}

src/Infra/ISession.cs

@@ -31,6 +31,6 @@ public interface ISession
     public string GetWindowHandle();
     public IEnumerable<string> GetWindowHandles();
     public IElement GetWindow(string windowId);
-    public string TakeScreenshot();
+    public string TakeScreenshot(int x, int y, int height, int width);
   }
 }

src/Infra/Session.cs

@@ -303,12 +303,16 @@ public IElement GetWindow(string windowId)
       }
     }
 
-    public string TakeScreenshot()
+    public string TakeScreenshot(int x, int y, int height, int width)
     {
-      using var bitmap = new Bitmap(1920, 1080);
+      x = Math.Max(0, x);
+      y = Math.Max(0, y);
+      height = Math.Max(1, height);
+      width = Math.Max(1, width);
+      using var bitmap = new Bitmap(width, height);
       using (var g = Graphics.FromImage(bitmap))
       {
-        g.CopyFromScreen(0, 0, 0, 0,
+        g.CopyFromScreen(x, y, 0, 0,
         bitmap.Size, CopyPixelOperation.SourceCopy);
       }
 

src/WinAppDriver/Controllers/SessionController.cs

@@ -330,7 +330,14 @@ public IActionResult TakeScreenshot(string sessionId)
       return ExecuteCommand(Command.TakeScreenshot, sessionId, null, null);
     }
 
-  [HttpPost]
+    [HttpGet]
+    [Route("{sessionId}/element/{elementId}/screenshot")]
+    public IActionResult TakeScreenshot(string sessionId, string elementId)
+    {
+      return ExecuteCommand(Command.TakeScreenshot, sessionId, null, elementId);
+    }
+
+    [HttpPost]
     [Route("{sessionId}/timeouts/async_script")]
     [Route("{sessionId}/url")]
     [Route("{sessionId}/forward")]
@@ -357,7 +364,6 @@ public IActionResult UnknownPost(string sessionId)
     }
 
     [HttpGet]
-    [Route("{sessionId}/element/{elementId}/screenshot")]
     [Route("{sessionId}/url")]
     [Route("{sessionId}/cookie")]
     [Route("{sessionId}/element/{id}/location_in_view")]