Highlight sources with unseen source submissions

[eloquence]

User Story

As a journalist, I want to know at a glance which sources have unseen submissions, so that I can quickly identify new information that may require attention.

Justification

Seen/unseen status is a fundamental paradigm in messaging applications. The feature exists in the pre-workstation user experience (as a global status of messages and documents, internally referenced as is_downloaded, and shown in the Journalist Interface UI as bolded/unbolded). User research has validated the need for similar functionality in the SecureDrop Client.

Mockup

(Design specification to come)

Terminology:

Source list: the panel on the left in the client view, which shows a list of sources by codename (e.g., "benign artichoke")

Conversation view: the panel on the right, which shows the messages, documents and replies for the currently selected source

Submissions: Messages or files sent by a SecureDrop source

Unseen submissions: Messages or files that have not been displayed in the conversation view of the SecureDrop Client, or downloaded via the Journalist Interface, by any user

Acceptance Criteria

Basic seen/unseen status and synchronization

Given that there are sources with unseen submissions on the server
When the client retrieves these submissions as part of a sync
And those sources are not actively selected
Then these sources should be displayed as unseen in the source list

Given that there is a source with unseen submissions on the client
And that all submissions from this source are marked as seen on the server
When the client synchronizes with the server
Then the source should be displayed as seen in the source list

Explanation: The information on the server is canonical; the same user may be connecting from more than one workstation.

Selecting a source with unseen submissions

Given that there is a source with unseen submissions on the client
When I select that source in the source list
Then the source should immediately be displayed as seen in the source list
And the client should record all the submissions from that source as seen by the logged in user

Explanation: Recording a change in the status implies a server-side API request followed by a client-side database update.

Error case

Given that there is a source with unseen submissions on the client
When I shut down the server while the client is running
And I select that source
Then the source should immediately be displayed as seen in the source list
And the client should keep retrying pending server operations indefinitely
And a generic network error should be displayed to the user until network access is restored

Explanation: We optimistically update the UI after the user selects a source, and keep retrying indefinitely if there's a network problem.

Offline usage

Given that there are sources with unseen submissions in the local client database
When I use the client in offline mode
Then all sources should be displayed in a read-only state (TBD) in the list source list
And seen/unseen status should not be modified as a result of my actions

Deleted journalist

Given that there is a source whose submissions have all been seen by a single journalist
When I delete that journalist user through the Admin Interface
Then the source should still be displayed as seen in the source list on the client after the next sync

Replies

Given that I have selected a source with unseen submissions and changed its status to seen
When I send a reply to that source
Then the source should continue to be displayed as seen in the source list

Explanation: Replies do not impact the seen/unseen status in the source list

Given that there is a source with no unseen submissions on the client
When another journalist sends a reply to that source
Then the source should continue to be displayed as seen in the source list after a sync

Explanation: As above, replies do not impact the seen/unseen status in the source list

Journalist Interface

Given that there is a source with unseen submissions on the server
When I initiate a download of these submissions via the web-based Journalist Interface
Then the source should be displayed as seen in the source list on the client after the next sync

Explanation: Downloading via the Journalist Interface counts as "seeing" a message or file.

Updated per user research review & team discussions - 2020-08-26 - @eloquence

[ninavizz]

I've died and gone to Gherkin heaven!

[eloquence]

Just a note that we've tentatively agreed that we'll aim for tracking this on a per-user basis from the start, instead of using the existing global read/unread flag (this information would be stored separately and independently on the server). Will rewrite the issue accordingly & file a corresponding server-side issue.

Update: As part of this change, I've also changed the language to new/seen/unseen, to set it apart from the language we use in the web UI, and to foreshadow additional "Seen by" functionality.

[redshiftzero]

This issue should address @creviera's note on #240 regarding the following situation:

[Unseen Submission]
Depending on how large the journalist's reply is, the source submission that was sent before the reply might not appear in the conversation box when the journalist refreshes. It'll appear in the middle of the conversation somewhere and that could be outside the scope of the conversation box.

[eloquence]

@redshiftzero @creviera @ninavizz

This issue has been reworded significantly over the last couple of days. Allie, as you make progress on your sprint commitments, I suggest we schedule a chat to go over this.

This issue is becoming quite complex; as we get a handle around the server-side changes that are required, we may discover a logical way to split it.

Regarding the reply issue Allie flagged in her review of #240, if I understand correctly, the issue is that replying will potentially scroll previously received messages/documents out of view. If so, I think that's fine and intended behavior, consistent with other messaging applications. I believe it's important, however, that we scroll new activity into view when you select the source to reply to, and that's reflected in the acceptance criteria above.

This "scroll new activity into view" behavior would only occur the first time; if you then un-select and re-select the source, it would scroll to the bottom. That's consistent, e.g., with the behavior in Slack or other applications designed to make new activity easily identifiable.

We've not yet fully user-tested the scrolling behavior and the "seen" indicator for files, so those are good areas to consider for the next round of testing as well.

[sssoleileraaa]

@eloquence - yes, the issue I was referring to can be compared to responding to a Slack message inline by starting a thread - sometimes it can be challenging to know where in the conversation a new message was added so they have an All Threads tab to see any changes that you might have missed.

In the case of SecureDrop, this is less likely to happen since message sync should happen every 5 minutes (and there isn't a threads feature). But it could happen when a journalist sends a bunch of replies or a long reply that pushes a lot of conversation history into backscroll around the same time that a source is submitting message(s) or a document(s).

Scrolling to the first occurrence of new activity the first time a source is selected helps indicate to the journalist that they should read from that point on, but it would also be useful (post beta) to have some sort of way to indicate which new submissions came in from the server that are now outside the scroll view for the scenario where it's not the first time a source is selected.

As you mentioned, the "seen" indicator for files could potentially address this issue. And we might want to consider adding unseen messages to the indicator.

[eloquence]

But it could happen when a journalist sends a bunch of replies or a long reply that pushes a lot of conversation history into backscroll around the same time that a source is submitting message(s) or a document(s).

Intuitively, my sense is that users would commonly notice without an additional notification (most messengers "de-escalate" notification levels for conversations that are in focus) due to the keyboard<->screen eye movement that is common during message composition for people who are not touch typists, and the screen focus that is typical for people who are. (Down the road if we can develop for accessibility, we'd want to make sure that a real-time voice readout occurs for blind users.)

I fully agree it's worth carefully thinking about the different options of no notification vs. ephemeral notification vs. persistent notification for this case.

This also raises the important general point of how the client should behave when a message or document arrives from a source while the source is selected. Assuming we can determine whether the client window is actively selected or not, I believe the behavior should likely be different -- i.e. we need to do more to get the user's attention if the client window is not selected but the source is. Will flesh out a bit along those lines.

(Update: Updated the spec a bit more to account for "currently selected" status. Whether the client window is selected or not adds significant complexity and I'm okay keeping that out of scope for now, assuming that the client is primarily used as a foreground application or not at all. But we should probably revisit that in an iteration.)

[ninavizz]

So: I want to be very, very careful with patterns we pick-up from other apps... and to inject a cautionary segue here, as I think it's very relevant to this discussion.

GMail and Slack are two app suites that target a broad enterprise audience. Unlike Microsoft, those teams are not led by MBA/Sales, but by UX-favoring product teams. So: while Microsoft's UX research efforts are very well funded and also target their broad enterprise base... Microsoft product teams have a ways to go, still, with embracing a "user first" approach in bridging the gap between research and what gets shipped. Microsoft products ship to meet the desire of their direct customers (IT executives, not the actual users) whereas Slack and Google "get" and embrace who actual users (and what their needs) are.

Messaging apps that are limited to the early-adopter crowd, hackers, techies, and security/privacy enthusiasts, are often not influenced by much user research. Signal has had an open "UX Lead" role on their Jobs board for almost 2 years now—so I doubt its product is influenced much from what is actually usable, vs what the developers chose to do. Discourse, I don't know much about... but the fact that it's a FOSS product whose team is mostly white dudes, and is a product I've only ever seen used for techie forums, has me doubting much research informs its shaping.

Slack, on the other hand, was created as very much the "early adopter crowd" app, but is now far more heavily influenced by usability research in what goes to market. Their micro-interactions are all very carefully studied, and I trust what both them and GMail put into their products, when in doubt, purely because of how I understand their research and product team dynamics (and investments in user research) to be. FYI.

[sssoleileraaa]

Just a note that we've tentatively agreed that we'll aim for tracking this on a per-user basis from the start, instead of using the existing global read/unread flag (this information would be stored separately and independently on the server). Will rewrite the issue accordingly & file a corresponding server-side issue.

Following up from a conversation I just had with @eloquence, we might also want to consider adding replies as new activity since they can come from different journalists signed in with different accounts.

[sssoleileraaa]

Also, bumping this again:

[Unseen Submission]
Depending on how large the journalist's reply is, the source submission that was sent before the reply might not appear in the conversation box when the journalist refreshes. It'll appear in the middle of the conversation somewhere and that could be outside the scope of the conversation box.

After discussing this more with @eloquence, rather than continue the current behavior where a source submission could be inserted in the middle of a conversation, we should always display the new activity at the bottom of the conversation view, even if that means we need to rename the file so it's ordered correctly. I will create an Issue that describes the desired behavior in more detail.

[ninavizz]

UX Spec: File Read

Checkmark SVG: https://drive.google.com/open?id=1uAFYxeQho-ebFJI0ulT4bpzWJu9yPxri

Per securedrop-ux/#39, the 'lil checkmark to the left of the filename will be the visual indicator. Ideally it will only turn-on, when the user clicks into a disp-VM that the file was opened within... but I suspect that's not possible to track w/in Qubes. If not, then I guess just turn it on 30 seconds after the user triggers the "View" action... assuming it doesn't crash Qubes? Heh.

[eloquence]

The plan of record, as discussed in the client sync meeting on 3/25, is that we may be able to initially build out this functionality as an (ephemeral) client-side feature. This will ensure that we can ship a version of this feature with the pilot, with a full client/server implementation as a stretch goal for the beta.

This would still require per-user storage of this data on the client, but all acceptance criteria related to the server state can be omitted for an initial implementation. I will create a stripped down set of acceptance criteria closer to implementation.

[sssoleileraaa]

Today, we discussed saving state before exiting the application so we know which source conversation to display when the application starts up again. More detail can be found here: #336

[ninavizz]

Blowing off some dust, and adding to #650

[eloquence]

We decided a while ago that this is Post-Beta (or possibly Mid-Beta), so moved off milestone & near-term backlog for now.

[eloquence]

These two components can be safely implemented independently:

• "new" (e.g., bolding in the source list)
• "seen" at the document level (e.g., checkmark at the document level)

To make this more manageable, I think we should split this issue accordingly, and prioritize them independently.

We will also need to decide if we want to do a client-side-only implementation at first, or immediately go with a client/server implementation. Regardless, we will have to account for multiple users using the same workstation. In other words, different sources will be "new" depending on who is logged in.

[sssoleileraaa]

When we were discussing this feature back and forth over a year ago (just reread to refresh my memory) the client was a lot different so some of the discussions earlier on were centered around old behavior like the 5-minute refresh cycle and the losing position in a conversation due to auto-scrolling. To make it easier to track/follow discussions around each feature (there is a lot of conversation not documented here, plus it's pretty easy to forget), I think we should split this up even further than what is suggested above. What do you think about this split @eloquence:

(the two @eloquence already mentioned)

• "new" (e.g., bolding in the source list)
• "seen" at the document level (e.g., checkmark at the document level)

(and...)

• new submissions/replies from other journalists for existing sources within the sourcelist (e.g. bolding a "New activity" indicator in the source widget vs bolding the source name for new sources)
• new submissions/replies from other journalists for existing sources within the conversation scrollarea (e.g. a line marker to indicate where the new activity starts)
• and a discussion issue/ google doc to talk about features that need a lot more discussion like a "seen by " feature or activity log etc.

We will also need to decide if we want to do a client-side-only implementation at first, or immediately go with a client/server implementation. Regardless, we will have to account for multiple users using the same workstation. In other words, different sources will be "new" depending on who is logged in.

I think client-side-only implementation to start off with makes sense. Once the server-side is ready, we can handle synchronization similar to the way we handle it with starring.

[eloquence]

Yeah, I'm in favor of a granular split into small improvements that can be delivered independently. Before we reorganize this issue, I'd like to get some input from our users, especially on the question if read/unread status should be global or per-user (as validated by having them play with prototypes that show different behaviors).

That could also influence the implementation strategy -- we do have a basic read/unread state on the server now (called downloaded on the server and exposed as is_read via the API, but only available for messages/submissions), so if we go with a "shared inbox / global state" model, there may be a relatively straightforward client/server implementation path.

[sssoleileraaa]

I agree we need more info. I think it would also be helpful to show a demo of what could happen when there are say two or three users on the same team working remotely at the same time.

[ninavizz]

@creviera All of the above need to be solved-for in a prototype for adequately testing all of this; as our users will have the same exact concerns. Which is why I'm cramping my brain trying to learn some React atm, so I can make more high-fidelity prototypes. Have also been thinking about how this could also make for easier ux/dev handoffs, and possibly even reduce... anyway, still trying to see if I can do this. :)

[sssoleileraaa]

@ninavizz I'd also be interested in going to these prototype sessions with the users to observe if you're cool with that, and I can help contribute code for prototyping if that helps. The global inbox scenario where there are multiple users using the shared inbox at the same time sounds pretty difficult to prototype, so if we want to switch to demo builds of the client, let me know.

[eloquence]

Here's where we currently are with this issue:

1. A first round of user research with pilot users has been completed. In this research, we're focusing on better understanding users' mental model of the SecureDrop inbox, whether read/unread state should be global or per-user, and how read/unread will interact with "newness" of sources.

2. We're reaching out to additional research participants from other organizations, and have updated to the prototype we use for user-testing to incorporate a "newness" indicator alongside read/unread status.

3. During the 8/6-8/20 sprint, we'll aim to complete additional user research (pending research participant availability) and to map out one or more implementation scenarios in more detail, to better understand server-side requirements which we may need to deliver as part of the SecureDrop 1.6.0 release (scheduled for 9/22).

Please consider the top-level description of this issue a placeholder until we've made a final decision on the scope of the first iteration, at which point we can update it, or split off sub-issues, as appropriate.

[eloquence]

Quick update:

• As committed in the last sprint, we've made great progress on technical design and scope for the MVP, as well as continued user research. User research is reaching a natural pause point.
• Further scoping discussion and review of user research to-date will happen early in in 8/20-9/2 sprint.
• Based on that, we should update the top-level issue description to hear to ensure it represents the canonical source of truth of what we'll implement, including the detailed acceptance criteria. We'll also need to map out the server-side requirements in a dedicated SecureDrop Core issue.
• We'll aim to implement server-side requirements (API changes, database migrations) in SD Core, in the 8/20-9/2 sprint. Timeboxed initially to 16 hours, may not get fully over the line. That said, we want to land those changes before feature freeze for 1.6.0 (9/8) if possible.

[eloquence]

Further scoping discussion and review of user research to-date will happen early in in 8/20-9/2 sprint.

We did this today, see meeting notes here (cross-linked from Client Wiki as well). We'll do a small work session tomorrow to update the acceptance criteria in this issue (and likely split off some chunks of it), and then we'll also have to add up-to-date design specifications.

[sssoleileraaa]

In addition, here is the Seen/Unseen V1 Spec which is a working document. You'll see a v2 doc linked to there to capture ideas that are not going to make it into v1, as we discussed today. We'll see what tomorrow's threat modelling session tomorrow yields. more to come!

[eloquence]

We've revised the acceptance criteria in this issue consistent with recent discussions & emerging team consensus. We've not decided the behavior in offline mode yet -- we may want to preserve display of the seen/unseen status without altering it, perhaps using a different style for source display & selection. @ninavizz will do some design explorations around that.

[eloquence]

Just a note that after review with @ninavizz I made one small addition to the acceptance criteria, for replies by another journalist that come in via a sync:

Given that there is a source with no unseen submissions on the client
When another journalist sends a reply to that source
Then the source should continue to be displayed as seen in the source list after a sync

This is just to make it clearer that journalist replies, whether by the logged in user or another user, will never cause a source to be displayed as unseen.

[eloquence]

Our expectation is that this feature will land after the major template consolidation release (which will include other unreleased changes in the SecureDrop Client, including reply badges). During the 10/15-10/28 sprint, @creviera will do timeboxed implementation work on a draft PR that implements the basic functionality outlined in this issue.

[ninavizz]

https://zpl.io/a8LL5eX

@creviera This Zeplin has the correct values for seen/unseen wrt font weights and coloration. The most notable things I can think of, are as follows:

• All text for an Unseen source, is #000
• All text and the paperclip for a Seen source, is #383838
• For selected source, the source's name is grimace blue and 600 weight Montserrat
• For Unseen sources, the source name and date are both either 700 or bold Montserrat
• For Unseen sources, the preview snippet is 600 SourceSansPro
• For Seen sources, the source name and date both 500 Montserrat
• For Seen Sources, the preview snippet is normal or 400 SourceSansPro

w00p w00p!

[eloquence]

(Removed from project board as #1165, which is tracked, is expected to fully resolve.)

[ninavizz]

@creviera Did the above design particulars make it into the linked PR?

[sssoleileraaa]

#187 (comment) lgtm! we can review the client together during a demo screenshare so you can see how it looks live.

[ninavizz]

Before doing a demo screenshare I'd first like to make sure the above specs are implemented into the code, so I can know what we're looking at. No rush! Just don't want it left behind with this Issue closed... so consider this more a flag for @eloquence. ;)