Support annotating image attachments

[michaelcfanning]

We have attachments today. A recent SARIF evaluator has noted that a screenshot which is associated with a result often comes with a bounding rectangle that highlights some area of interest in the screenshot. This bounding rectangle is the equivalent of a region associated with a source file. Probably this should be an array of bounding rectangles.

We might consider adding a region and/or set of annotations as well, so that other kinds of attachments could be associated with selected internal areas.

[ghost]

This is fascinating.

[michaelcfanning]

@lgolding Updated title with actual suggestion. I have more offline details on this SARIF producer I can provide.

[michaelcfanning]

@lgolding request this for CSD1 as we have a committed SARIF v2 producer who has requested the feature

[ghost]

@michaelcfanning Yes, I'll do it in CSD.1 and write it this weekend.

[ghost]

@michaelcfanning Yes, please send along details of the SARIF producer.

[ghost]

Proposal: Add regionAnnotations to attachment. Add screenshots to result. Invent supporting objects as shown.

attachment
  =description:message
  =fileLocation
  +regionAnnotations:regionAnnotation[] 

result
  =attachments:attachment[]
  +screenshots:screenshot[]

+regionAnnotation
  region
  message
 
+screenshot
  description:message
  fileLocation
  areaAnnotations:areaAnnotation[]
 
+areaAnnotation
  rectangle
  message
 
+rectangle
  top:integer, left:integer, bottom:integer, right:integer

[ghost]

@michaelcfanning Said ok in email.

You know, there's an alternative: Give region a message property. Then any region can be "annotated", the regionAnnotation object in my proposal goes away, and we have instead:

attachment
  =description:message
  =fileLocation
  +regions:region[] 

result
  =attachments:attachment[]
  +screenshots:screenshot[]

region
  +message
 
+screenshot
  description:message
  fileLocation
  areaAnnotations:areaAnnotation[]
 
+areaAnnotation
  rectangle
  message
 
+rectangle
  top:integer, left:integer, bottom:integer, right:integer

[ghost]

@michaelcfanning Even better (IMO): give rectangle a message and dispense with areaAnnotation:

attachment
  =description:message
  =fileLocation
  +regions:region[] 

result
  =attachments:attachment[]
  +screenshots:screenshot[]

region
  +message
 
+screenshot
  description:message
  fileLocation
  recangles:rectangle[]
 
+rectangle
  top:integer, left:integer, bottom:integer, right:integer
  message

We move towards a design philosophy where many objects have messages, just like many objects have property bags.

[michaelcfanning]

i am concerned about adding message to region. why aren't we utilizing the annotations objects here? the encapsulate a physical location, which can encapsulate regions. in this use, only the region information of the PLC may be populated.

also, what if we allowed bounding rectangles for attachments, just noting that they are only relevant if the attachment is an image? Leaving the following reduced proposal:

attachment
=description:message
=fileLocation
+annotations
+recangles:rectangle[]

result
=attachments:attachment[]

+rectangle
top:integer, left:integer, bottom:integer, right:integer
message

[ghost]

I like your proposal. I didn't use annotation because all the annotations on a screenshot referred to the same file. I didn't want to verify that all the fileLocations in the annotations referred to the same file.

But your way is simpler. I'll send it out. This eliminates the screenshot value for role, which I also like. Not every image of interest is a screenshot, but any image might benefit from annotating like this.

[ghost]

@michaelcfanning I'm still not wild about using the annotation object when you already know the fileLocation and just need the region. Yes, there are one or two other places in the spec where we say things like "you have to mention the URL unless you're inside a file object that already mentions the URL" but I've never liked it.

What's your objection to region.message?

It would be symmetric with rectangle holding its own message, and it would make attachment symmetric between regions and rectangles:

attachment
  =description:message
  =fileLocation
  +regions:region[]
  +rectangles:rectangle[]

[michaelcfanning]

@lgolding: why does physical location need a message if region has a message?

[ghost]

@michaelcfanning Not every physicalLocation has a region. Some just have a fileLocation.

[michaelcfanning]

what's the distinction between physicalLocation.message and physicalLocation.region.message if a viewer finds that both have been populated?

[ghost]

It could use region.message as hover-text (which, remember, was the original motivation for annotation).

[ghost]

It's the same use case we imagine for rectangle.message, right?

[michaelcfanning]

ok, i think i buy it.

[ghost]

@michaelcfanning Retitled to match the design we settled on.