New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Screen Annotations: graphically mark events and add other interessting annotations to Screenshots #149

Closed
adiherzog opened this Issue Mar 28, 2014 · 40 comments

Comments

Projects
None yet
4 participants
@adiherzog
Member

adiherzog commented Mar 28, 2014

As a user of the scenarioo documentation I want to see important informations directly inside the screenshots as so called "screen annotations", such that I can see where the test interacted with the UI or other important informations that we want to highlight in the screenshots (e.g. caption/subtitle, URL the test entered or changed, click region, typed text, selection items inside a dropdown, mouse-over-texts, validated fields, etc.)

We identified the following important types of annotations so far, that we want to support with our design

  • click events (button, checkbox, etc.)
  • select event inside a combo-box or list
  • typing inside a text box
  • typing an URL in the browser navigation address bar
  • pressing refresh / back / forward buttons in browser
  • mark a field who's content was validated by the test (assertion)
  • mark fields for additional informations about the fields (e.g. all selection items inside a combo box)
  • any important text you want to put visually on the screenshot (any warning, or a caption/subtitle about what happens in this step)
  • there might be more types of annotations we did not think of yet

In general the annotations are realized generically, such that any other information can be stored in a similar way. The design therefore consists of the possibility to place arbitrary rectangular regions as "screen annotations" on a step, with addtional informations like text and description and additional informations (as generic data structure as other details in scenarioo documentation format). Detailed format description see acceptance criterias and JSON example below.

Acceptance criteria

  • The API (Java and C#) is extended such that multiple "screen annotations" can be added to a step. Each screenAnnotation consists of following informations (see also JSON example below):
    • region: rectangular area on the screen to highlight (x,y,width,height), origin of coordinates is the top left corner of the screenshot.
    • screenText (optional): a text to display inside the region directly on the screen (client will cut off the text if too long). The text is also displayed in popup (in full length, in any case).
    • title (optional): a title to display in the info popup on an annotation as bold title (if not set, 'screenText' is used as popup title, or 'description')
    • description (optional): A longer description text to only display inside the popup that appears when clicking on an annotation (below the other 'text' which is displayed as strong caption above the description text inside the popup)
    • style (optional, default="DEFAULT"): Determines, how the marked region is styled. One of several predefined styles: "CLICK" | "KEYBOARD" | "EXPECTED" | "NAVIGATE_TO_URL" | "WARN" | "ERROR" | "INFO" | "HIGHLIGHT" | "DEFAULT".
    • clickAction (optional): Determines what happens if the annotated region is clicked. One of "TO_NEXT_STEP" | "TO_URL". Default behaviour is to open the information popup.
    • clickActionUrl (optional): URL that is opened in case the clickAction is set to "TO_URL". (mandatory if clickAction is set to "TO_URL")
    • clickActionText (optional): information to be displayed as mouse over for an attached click action (if not avaialble a generic text "go to next page" or "open " will be displayed.)
    • details (optional): Generic Metadata as everywhere else in Scenarioo. This is also displayed in the popup below the description (usual rendering as in metadata trees on other pages of Scenarioo).
  • Whenever a step contains screen annotations, the Scenarioo client displays them in the correct position as an overlay on the screenshot (rectangular marker, icon dependent on style, text inside rectangular region).
  • the regions are placed correctly over the screenshot in all supported browsers and are placed correctly depending on the size of the screenshot.
  • the "screenText" is displayed inside an annotation if available, if regions get to small the "screenText" will not appear anymore (user can still click to open the details popup to see it).
  • each annotation has an icon in the upper right corner, that is different depending on its style (if no style is specified, "DEFAULT" is used)
  • clicking on the icon of an annotation opens a popup to display information about the annotation (text, description, details)
    • the popup displays a title as bold text (property 'title' or "screenText" or otherwise the same as "description")
    • below the title the 'description' is displayed (if not allready same as title)
    • if a clickAction is defined, it is displayed as a Link below the description (with text 'clickActionText' or a generic text)
    • all details are displayed (same as other metadata)
    • the popup is closed when clicking outside the popup
    • the popup can be scrolled (if many details are contained, using browsers scroll bar)
  • if no clickAction is specified, clicking the whole rectangular region has the same effect as clicking the image icon button to open the popup
  • If a screen annotation has a defined clickAction
    • the region is clickable with the specified behaviour (go to next step, open a specified clickActionURL in new window).
    • the annotation is styled as a special link with a mouse over tooltip that informs the user about the attached userAction (before clicking)
  • button "show/hide annotations": user can choose to turn rendering of screen annotations on and off in the client on every step
    • this button is only displayed on steps that have annotations
    • this setting is stored in local storage in the client, if a user turned it off, all annotatons will disappear, until the user turns it on again
  • all objects stored in details of screen annotations are also stored in object repository and can therefore be retrieved (see new demo tab "UI Elements" on home page)

JSON structure on step object for the client

Only the "region" field is mandatory in the API.

{
    "stepStatistics" : { ... },
    "stepIdentifier" : { ... },
    "fallback" : false,
    "step" : {
        "page" : { ... },
        "metadata" : { ... },
        "html" : { ... },
        "screenAnnotations" : [
            {
                region : {
                    x: 123,
                    y: 222,
                    width: 222,
                    height: 20                  
                },
                screenText: "Blabla",
                                    title: "Title for Popup",
                description: "Bliblubb",
                style: "CLICK" | "KEYBOARD" | "EXPECTED" | "NAVIGATE_TO_URL" | "WARN" | "ERROR" | "INFO" | "HIGHLIGHT" | "DEFAULT",
                clickAction: "TO_NEXT_STEP" | "TO_URL",
                clickActionUrl: "http://...",
                                    clickActionText: "Open documentation 'XY'"
                details: {
                    ...
                }
            }
        ],
        "stepDescription" : { ... }
    },
    "stepNavigation" : { ... },
    "useCaseLabels" : { ... },
    "scenarioLabels" : { ... }
}

limitations (or smaller changes regarding the original design):

  • all annotations are currently styled in same color (orange), only the icons are different between styles. also the text sizes are the same for all annotation styles, later we could improve this to make this customizable. The border color currently depends on whether the annotation has a clickAction or not.
  • all annotations currently only support text without braking to more than one line (overflow of too long texts does not work properly otherwise, see https://css-tricks.com/almanac/properties/t/text-overflow/ ---> following described solution could help for that to work: http://www.mobify.com/blog/multiline-ellipsis-in-pure-css/ but I did not get this to work)
  • there is no close button for the details dialog, user can close it by clicking outside the dialog

see also

This design also covers story #393 because it is generic and extendable enough to cover this additional possibility to add arbitrary annotations.

@adiherzog adiherzog added this to the backlog-prio1 milestone Mar 28, 2014

@adiherzog adiherzog self-assigned this Mar 28, 2014

@danielsuter

This comment has been minimized.

Show comment
Hide comment
@danielsuter

danielsuter Apr 8, 2014

Member

Can you elaborate on this issue? E.g. what's the purpose of this information, how will it be created / stored?

Member

danielsuter commented Apr 8, 2014

Can you elaborate on this issue? E.g. what's the purpose of this information, how will it be created / stored?

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Apr 8, 2014

Member

I just updated the issue. I'd like to get some feedback from the Scenarioo developers whether everybody is happy with the idea. Some open questions:

  • Should one step allow multiple marked regions or only one?
  • Should a marked region also have information about whether it is a click or a value change event that generated the information?
Member

adiherzog commented Apr 8, 2014

I just updated the issue. I'd like to get some feedback from the Scenarioo developers whether everybody is happy with the idea. Some open questions:

  • Should one step allow multiple marked regions or only one?
  • Should a marked region also have information about whether it is a click or a value change event that generated the information?
@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Apr 8, 2014

Member

I would allow for multiple markers for one image, if easily feasable.
Or are there any arguments for not allowing multiple?

Member

bruderol commented Apr 8, 2014

I would allow for multiple markers for one image, if easily feasable.
Or are there any arguments for not allowing multiple?

@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Apr 8, 2014

Member

If you give the possibility to simply attach a ObjectDescription onto a marker (like you can put them into details of any other objects in the scenarioo documentation), then the users are able again to put their whatever application-dependent informations onto a marker.

Advantage of this solution:
Markers could have some special event attached (modeled as an ObjectDescription or ObjectReference of type "event" or "click-event", "value change event" etc.), and because of the generic object repository you can later (as soon as these views are implemented) search for such specific events.
This could be very helpful, powerful and flexible.

Just an idea, please think about it, when refining the concept.

Member

bruderol commented Apr 8, 2014

If you give the possibility to simply attach a ObjectDescription onto a marker (like you can put them into details of any other objects in the scenarioo documentation), then the users are able again to put their whatever application-dependent informations onto a marker.

Advantage of this solution:
Markers could have some special event attached (modeled as an ObjectDescription or ObjectReference of type "event" or "click-event", "value change event" etc.), and because of the generic object repository you can later (as soon as these views are implemented) search for such specific events.
This could be very helpful, powerful and flexible.

Just an idea, please think about it, when refining the concept.

@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Apr 8, 2014

Member

Some more feedback to current described solution:

  1. I would add a field "style-class" to each marker, such that users can define their own marker styles to apply, this is open for additional types of markers, not only rectangle and point.
    But maybe this makes only sense as soon as #148 is implemented. But why not already supporting the setting of a custom style class (and just deferring the customization of the visualization of those classes to the story #148 )?
    This may also solve your question for different type of events?
    You could use a different style class for value change than for click, and later customize this visualization.
  2. If point 1 is realized I do not see why we should have a distinction of two fixed types of markers (rectangle and point) anymore? I think one default styling as rectangular is sufficient for most applications. Those needing a different styling as circular point can introduce it by customization described in point 1. Or we could support 2 predefined custom styles, and users can add additional ones later, as soon as they need.
  3. Tooltips on Mouse over: maybe not the best solution in case you want to allow to use Scenarioo also on Tablets? A popup on click would be nicer maybe, but then the navigation to the link needs to be designed differently.
  4. I think one additional link-type could be helpful: "link-to-url" together with an additional field url, when clicking on the marker, the URL is opened (could be any wiki page with additional information about the marked region).

Ok, just my thoughts. you can of course put some of these to another story for marker API version 2.0 ;-) But maybe some of these thoughts are worth to consider for version 1?
What do you think?

Member

bruderol commented Apr 8, 2014

Some more feedback to current described solution:

  1. I would add a field "style-class" to each marker, such that users can define their own marker styles to apply, this is open for additional types of markers, not only rectangle and point.
    But maybe this makes only sense as soon as #148 is implemented. But why not already supporting the setting of a custom style class (and just deferring the customization of the visualization of those classes to the story #148 )?
    This may also solve your question for different type of events?
    You could use a different style class for value change than for click, and later customize this visualization.
  2. If point 1 is realized I do not see why we should have a distinction of two fixed types of markers (rectangle and point) anymore? I think one default styling as rectangular is sufficient for most applications. Those needing a different styling as circular point can introduce it by customization described in point 1. Or we could support 2 predefined custom styles, and users can add additional ones later, as soon as they need.
  3. Tooltips on Mouse over: maybe not the best solution in case you want to allow to use Scenarioo also on Tablets? A popup on click would be nicer maybe, but then the navigation to the link needs to be designed differently.
  4. I think one additional link-type could be helpful: "link-to-url" together with an additional field url, when clicking on the marker, the URL is opened (could be any wiki page with additional information about the marked region).

Ok, just my thoughts. you can of course put some of these to another story for marker API version 2.0 ;-) But maybe some of these thoughts are worth to consider for version 1?
What do you think?

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Apr 9, 2014

Member

Thanks @bruderol for your input. I changed the acceptance criteria a bit. Here are my decisions:

  • We can allow multiple markers, as this should be easily implementable.
  • I added a new field "event", so that we can make an event-dependant styling (e.g. show a mouse pointer in case of a click event).
  • There's no field "style-class", as this should be added when we actually implement #148. If we never implement issue #148, then the custom style-class is just waste. I'd like to keep it clean.
  • The point type is different to the rectangle in that it only needs the x / y coordinates, but no width or height. Therefore I kept this distinction.
  • A popup for the description is not feasible because then we could not have the "go to next step" link anymore, which has a higher priority in my opinion. Scenarioo is not tablet friendly or fully responsive at the moment as we use tooltips in different places. We could open a new issue for solving this in general.
  • "link-to-url" is a good idea, but I think we should add it only when somebody is actually gonna use it.

@bruderol @xeronimus @danielsuter @gitu @mautechr Please review the acceptance criteria again and let me know whether something is missing, that you would like to actually use in your current projects. I'd like to implement only the minimal set of features, which we currently need (YAGNI principle). If somebody needs more in the future, it's easy to add more features later.

Member

adiherzog commented Apr 9, 2014

Thanks @bruderol for your input. I changed the acceptance criteria a bit. Here are my decisions:

  • We can allow multiple markers, as this should be easily implementable.
  • I added a new field "event", so that we can make an event-dependant styling (e.g. show a mouse pointer in case of a click event).
  • There's no field "style-class", as this should be added when we actually implement #148. If we never implement issue #148, then the custom style-class is just waste. I'd like to keep it clean.
  • The point type is different to the rectangle in that it only needs the x / y coordinates, but no width or height. Therefore I kept this distinction.
  • A popup for the description is not feasible because then we could not have the "go to next step" link anymore, which has a higher priority in my opinion. Scenarioo is not tablet friendly or fully responsive at the moment as we use tooltips in different places. We could open a new issue for solving this in general.
  • "link-to-url" is a good idea, but I think we should add it only when somebody is actually gonna use it.

@bruderol @xeronimus @danielsuter @gitu @mautechr Please review the acceptance criteria again and let me know whether something is missing, that you would like to actually use in your current projects. I'd like to implement only the minimal set of features, which we currently need (YAGNI principle). If somebody needs more in the future, it's easy to add more features later.

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Apr 9, 2014

Member

I simplified the API extension further by removing points. Daniel and I discussed this today and we don't see, why a point should be marked. At least in Selenium you can always get the with and the height of the element that is to be clicked or edited. Does anybody disagree? Maybe somebody that is not using Selenium?

Member

adiherzog commented Apr 9, 2014

I simplified the API extension further by removing points. Daniel and I discussed this today and we don't see, why a point should be marked. At least in Selenium you can always get the with and the height of the element that is to be clicked or edited. Does anybody disagree? Maybe somebody that is not using Selenium?

@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Apr 9, 2014

Member

In case somebody realy needs a point he can still use a rectangular marker to put a quadrat with a small width to mark this point. And later when custom style is supported he could even visualize it using its own style for such points, e.g. to mark it with a small circle.

Therefore I also think this is sufficient.
No need for points.

Member

bruderol commented Apr 9, 2014

In case somebody realy needs a point he can still use a rectangular marker to put a quadrat with a small width to mark this point. And later when custom style is supported he could even visualize it using its own style for such points, e.g. to mark it with a small circle.

Therefore I also think this is sufficient.
No need for points.

@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Apr 9, 2014

Member

I think in general this is very well designed. Very good job. 👍

Sounds like ready to implement :-)

I still miss some advanced features (like linking to an application specific ObjectDescription that provides more information about the event from the marker), but I can put that into a separate story, no problem. This advanced feature needs some more detailed design anyway.

And I absolutely agree: only build what is realy needed for sure in the first version and what we agree on. We still can extend in small steps.

Member

bruderol commented Apr 9, 2014

I think in general this is very well designed. Very good job. 👍

Sounds like ready to implement :-)

I still miss some advanced features (like linking to an application specific ObjectDescription that provides more information about the event from the marker), but I can put that into a separate story, no problem. This advanced feature needs some more detailed design anyway.

And I absolutely agree: only build what is realy needed for sure in the first version and what we agree on. We still can extend in small steps.

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Apr 9, 2014

Member

Thanks for the positive feedback. I'm looking forward to seeing this feature in action 😎

Member

adiherzog commented Apr 9, 2014

Thanks for the positive feedback. I'm looking forward to seeing this feature in action 😎

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Dec 5, 2014

Member

I added the needs sponsor label because this story is a real need for some customers.

Member

adiherzog commented Dec 5, 2014

I added the needs sponsor label because this story is a real need for some customers.

@adiherzog adiherzog removed their assignment Dec 5, 2014

@adiherzog adiherzog modified the milestones: Coding Days 2015 Candidates, Backlog Prio 1 Dec 5, 2014

@danielsuter

This comment has been minimized.

Show comment
Hide comment
@danielsuter

danielsuter Dec 10, 2014

Member

Will we only implement it, if we find a sponsor or could it be something we do anyway?

Member

danielsuter commented Dec 10, 2014

Will we only implement it, if we find a sponsor or could it be something we do anyway?

@adiherzog

This comment has been minimized.

Show comment
Hide comment
@adiherzog

adiherzog Dec 10, 2014

Member

We can do it anyway, but it would be nice to have a sponsor.

Member

adiherzog commented Dec 10, 2014

We can do it anyway, but it would be nice to have a sponsor.

@bruderol

This comment has been minimized.

Show comment
Hide comment
@bruderol

bruderol Dec 10, 2014

Member

The rule is: If you like to spend your education or spear time for it, you can. It's your decission!

But we should be aware, that for features that are marked like this, we see a high chance, that somebody could sponsor it. and therefore we should better try hard to find a sponsor for things like that, such that we can spend our education time on other important things.

Member

bruderol commented Dec 10, 2014

The rule is: If you like to spend your education or spear time for it, you can. It's your decission!

But we should be aware, that for features that are marked like this, we see a high chance, that somebody could sponsor it. and therefore we should better try hard to find a sponsor for things like that, such that we can spend our education time on other important things.

@bruderol bruderol changed the title from Mark screenshots to Mark click events on Screenshots Mar 30, 2015

@bruderol bruderol changed the title from Mark click events on Screenshots to Mark click or change events on Screenshots Mar 30, 2015

@adiherzog adiherzog self-assigned this Jun 15, 2015

adiherzog added a commit that referenced this issue Feb 3, 2017

adiherzog added a commit that referenced this issue Feb 3, 2017

adiherzog added a commit that referenced this issue Feb 3, 2017

adiherzog added a commit that referenced this issue Feb 3, 2017

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
Introduced popup window for annotation details.
Forced all modals to close when location changes.
Separate annotation fields for screen text and title (title only displayed in popup).
Introduced service for annotation logic (e.g. mapping of styles to icons, which is used in two controllers)
Introduced one package for all annotation related components.

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
fixed lint violation and broken update of annotation positions on load.

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
fixed lint violation

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
defined new module for screen annotations (preparation for later modularization in vertical functionalities)

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
fixed e2e test and refactored e2e tests a little (start of application with popup is not same use case as "list use cases")

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
fixed lint error

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
only show button for showing/hiding annotations when annotations are present
(and refactored screen annotation button directive to John Pappa Style guide)

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
click actions and styling of annotations with click actions and mouse pointer for popups

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
additional test data in demo-data for annotations of all styles, with and without click actions, also with clickActionText

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
- tooltip for annotations with click action
- styling improvements: color/margin for text in annotations, documented non-working overflow of text for multiline text boxes

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
Display click action also in info popup

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
adjusted webtests to additional test in demo data

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
adjusted position of annotations in demo data

adiherzog pushed a commit that referenced this issue Feb 3, 2017

#149 Screen Annotations:
fixed mouse pointer on click action link in popup
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment