Design support for external bibliography service

[blalbrit]

Requirement for Parker:

• bibliography entries maintained in an external service
• bibl entries tagged to associate them with relevant mss. in Parker
• Spotlight needs to be able to query the external service and build a bibliography view within the item view

Sub-tasks:

• UX design
• technical requirements

[ggeisler]

Below is a first pass at how we might do this, intended to start a discussion of some of the details and assumptions we'll need to decide on before this is an actionable ticket.

The first assumption is that there is a service endpoint that accepts an object druid and returns a set of bibliography references that match that druid. So when a visitor in an exhibit navigates to an item show page:

• If the exhibit is configured with a valid service endpoint, druid is sent to the service
• If the endpoint returns a non-empty set, the returned set of items is parsed and displayed as list items in a named section at the bottom of the item details page

Generically, something like this:

Where the configuration might look something like:

And the populated item details page like this, with the Bibliography section in a new row below the metadata (Parker items have very long lists of refs, so this feels like the best way to show them):

Other assumptions

The configuration mockup above assumes the service is a bibliography service and the key value sent to the service is a druid, but by requiring the exhibit admin to enter the service endpoint (and allowing them to override the section heading) I believe we'd be leaving it generic enough that exhibits other than Parker could theoretically set up their own bibliography service endpoints and use the feature. So it's not Parker-specific but it is bibliography specific (or at least, service that provides a set of items we're going to display as a list specific).

For Parker and other exhibits with item bibliographies the above assumption is probably fine, but thinking more broadly: do we want to make it even more generic? Potentially this could just be an "Item Details page" service and we display whatever is returned from the endpoint. But that would require a more complex configuration, such as what HTML tags to wrap the returned values in, how to sort them, whether what's returned is even a set of items versus a single item, etc.

Although the mockup doesn't show it, ideally the bib service would store the references HTML-encoded, so we retain the proper reference formatting. The service should also be able to return the set of refs sorted alphabetically.

[blalbrit]

If we're going to go with Zotero for the bibliography service, see:
https://www.zotero.org/support/dev/web_api/v3/basics
and
https://library.brown.edu/cds/2015/06/29/zotero-api-web/

I believe the simple return for a formatted bibliography list caps at 150 objects, which might be a problem for some of the objects in this collection. I believe, though, we can get a larger return if we just take json back - which would then require formatting in Spotlight, but would get us around the issue of number-of-item caps.

[drh-stanford]

I dug into the API and below is some rough code that might do what we want -- assuming we have a Zotero collection to export. And the 150 item limit is correct and it's unlimited if we just want the data and do the styling ourselves using a 3rd party gem (CAP does this -- see https://github.com/sul-dlss/sul_pub/blob/master/Gemfile#L23-L24).

In this example, I'm using a Zotero collection -- which has a specific identifier that Zotero generates. The default sorting appears to be Author-Date.

puts '<ul>'
url = "https://api.zotero.org/users/#{USER_ID}/collections/#{COLLECTION_ID}/items" + '?' +
      URI.encode_www_form(v: 3,
                          format: :json,
                          include: 'bib',
                          key: API_KEY,
                          start: 0,
                          limit: LIMIT)
open(url) do |conn|
  items = JSON.parse(conn.read)
  puts items.collect {|item| '<li>' + item['bib'] + '</li>'}.join
end
puts '</ul>'

which outputs the following using my test data:

<ul>
<li><div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Allen, John. <i>Another Book</i>, 2017.</div>
</div></li>
<li><div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Doe, Jane. &#x201C;My Article.&#x201D; <i>JXXX</i> 1, no. 2 (2016): 3&#x2013;4.</div>
</div></li>
<li><div class="csl-bib-body" style="line-height: 1.35; padding-left: 2em; text-indent:-2em;">
  <div class="csl-entry">Doe, Jane. <i>My Book</i>, 2017.</div>
</div></li>
</ul>

To map the Zotero collections into a druid, we may need to have the user enter the Zotero collection key (or possibly the name). Otherwise, they could name the Zotero collection using the same title as the druid object -- we could automate the mapping that way. Either way, I think the user will need to do something to provide hints/data for the mapping.

[drh-stanford]

Open Issues

1. How does the exhibit user associate an item (key = druid) to a Zotero Collection?
   • They could name the Zotero collection after the druid (or maybe use the exact title if Zotero support the same naming conventions)
   • They could drop the Zotero collection URL into a Spotlight Item field
   • Or, ...?
2. How does the exhibit user authenticate into Zotero? Or do we force exhibit users to make their bibliographies public as authentication is not required for read access to public libraries. Alternatively we need to have their Zotero API key (and possibly need to do OAuth).
3. How do we handle Zotero API outages? e.g., we're seeing 503 errors today on their API.
4. Do we support Zotero "groups" or only "users"?
5. Do we support Zotero's nested collections and nested items data model?
6. What specific citation style and sorting do we want to use? The default is Zotero's "chicago-note-bibliography" that sorts by Author-Date.
7. Is the Zotero limit of a 150 bibliography entries per exhibit item sufficient? If not, we will need to do our own styling.

User Data Needed

1. The "Service Endpoint" would always be https://api.zotero.org/users/#{USER_ID} so the data requires is the Zotero user id. To get this id the exhibit user will need to copy it from the Zotero "API" section of the "Settings" interface.

This is how to make the (entire) library public on the Zotero dashboard, and the "Feeds/API" tab is where you can copy the user id:

Bibliography service

We will use Zotero as the bibliography database, and they have a public Web API (though I couldn't find an existing Ruby gem that supports it). Note that there's a 150 item limit per Zotero collection if we use their styling engine. Here's an example API call to get all of the items from a Zotero collection output as HTML with styling: https://api.zotero.org/users/475425/collections/9KH9TNSJ/items?format=bib

Technical integration design into Spotlight

TBD -- partially blocked on some answers from the open issues

[blalbrit]

In checking the current data, many of the Parker objects have a bibliography in excess of 150 objects, so I think we'll need to pull the raw JSON back (which allows more than 150 objects). That gives us the opportunity to determine our style output more freely.

I would be fine with using the same styling we currently use in Parker on the Web (see here for exx. https://parker.stanford.edu/parker/actions/bibliography_search_results.do?
journal=&primarySourceRef=286&author=&title=&sort=alphabetic&shortTitle=&comment=&date=&referenceTypes=) less the comments/reproductions links.

[blalbrit]

For Parker, specifically, for:
How does the exhibit user authenticate into Zotero? Or do we force exhibit users to make their bibliographies public as authentication is not required for read access to public libraries. Alternatively we need to have their Zotero API key (and possibly need to do OAuth).

This will be a public bibliography.

If we want to generalize this for general Spotlight, this could be revisited - but the requirement for Parker is simply to support a public collection.

[blalbrit]

How do we handle Zotero API outages? e.g., we're seeing 503 errors today on their API.

I think a simple user-message would be acceptable on a 503 "The bibliography service is not currently available" or something similar. @ggeisler - do you have preferred wording for something like this?

I assume placement would be on the item-view page in the section where the bibliography would normally appear.

[blalbrit]

What specific citation style and sorting do we want to use? The default is Zotero's "chicago-note-bibliography" that sorts by Author-Date.

See above for style.

For sorting, Author-Date is perfect.

[drh-stanford]

Re: style. I don't recognize that bibliography style in the mockup. Does it have a name, or is it a custom style for the Parker community? Ideally, for the technology stack, the style we want has a CSL implementation available in their repository. Right now it looks like Parker is doing an entirely custom Java-based styling (see https://github.com/sul-dlss/parker-webapp/blob/acfe1a98d6d39bd3a044606344882dff85977b4e/WEB-INF/src/edu/stanford/sulair/parker/search/BibliographySearchResult.java)

[blalbrit]

Yes - the Parker style was tricky as we were going between British style guides and American. Let's go with Chicago now (author-date).

[drh-stanford]

For the open issue on how to associate exhibit items with bibliography items, we can use Zotero tags. The exhibit user would need to add the druid to the tag list for all the items.

You can search the library by TAG like so: https://api.zotero.org/users/3802090/items?tag=druid:iouoiuoiuo

[drh-stanford]

Ok, I've written up a technical requirements doc here -- https://gist.github.com/drh-stanford/ffa984205f406d6267e5fa8e2606ee68. This needs some review...

[ggeisler]

@blalbrit Regarding your earlier suggestion about wording when the Zotero API is not available, your suggestion of "The bibliography service is not currently available." sounds fine to me.

[drh-stanford]

Talked with @blalbrit and @mejackreed and we're going to investigate a "Sync with Zotero" workflow that would be in the Exhibits Settings that the curator updates regularly. This simplifies the implementation somewhat and avoids the delay when viewing exhibit items.

[drh-stanford]

I've updated that requirements doc to use the "Sync with Zotero" workflow. We need some UX updates to implement it. Note that I've also added a couple more open issues. https://gist.github.com/drh-stanford/ffa984205f406d6267e5fa8e2606ee68

[ggeisler]

@drh-stanford @blalbrit @mejackreed @jkeck Updated mockups below. Let me know if you think I'm missing anything or if you have concerns or suggestions.

All mockups below reflect the addition of a "Synchronization status" section at the bottom of the form to deal with the synchronization requirement. The rest of the form is unchanged from my previous mockup described in an earlier comment.

First-time configuration

If the exhibit has not yet been successfully configured with a Zotero library, the Synchronization status section should show a text message that reflects the never synchronized state and hide the synchronization button (since it has no utility until the configuration is done).

After successful configuration and synchronization

I assume that when the form is saved with a valid configuration values, synchronization will immediately begin. We could display a flash message after the Zotero ID and group/user type selections have been confirmed as valid to let the user know what's going on:

For this feature, I think that the flash message is probably sufficient for providing status (versus a more elaborate status display to provide counts of items synchronized, etc., such as we use for indexing item status) but that could be open for debate.

Subsequent visits to the Services > Bibliography page

When the admin returns to the Services > Bibliography page after an initial, successful Zotero synchronization, we provide a different text message with the last sync date, and add a synchronize button with some associated text at the bottom of the form:

(the sync button uses the .btn-default .btn-sm classes for styling)

When the admin clicks the Synchronize with Zotero button, assuming the configuration hasn't changed and there are no errors related to connecting with the service, we'd again show the flash message for status:

[drh-stanford]

@ggeisler @blalbrit we have a few questions about the UX...

1. What happens if the sync fails? The sync is happening in a background job and it's likely that the job will take ~10 mins or so.
2. The last screenshot should have a Zotero ID in it -- our understanding is that it the only way to get the Sync button to show (to have an ID and to have had a successful run).
3. It's a little confusing to have the Sync button and the Save Changes buttons so close. Can we somehow put some distance to the "status" information -- like a different section of the page, for example?
4. How does the user know when the sync is completed successfully? Is it just that status line text?
5. What does Save Changes do if the Zotero ID has changed? WIll it rebuild everything against a new Zotero library, right?

[ggeisler]

@drh-stanford (@blalbrit feel free to chime in if you disagree with any of my responses): I'll answer the easy ones first:

The last screenshot should have a Zotero ID in it -- our understanding is that it the only way to get the Sync button to show (to have an ID and to have had a successful run).

If you mean the Zotero ID should be showing in the input field, yes, I agree. I thought of that after posting the comment, but had already deleted my session where I was doing the mockup. I meant to edit the comment to mention it but forgot.

It's a little confusing to have the Sync button and the Save Changes buttons so close. Can we somehow put some distance to the "status" information -- like a different section of the page, for example?

I'm not sure I agree with the button placement being confusing. The Save changes button is in the same location on every page and the admin user is likely used to it being in that location. The synchronize button is a different color, aligned left versus the Save changes button being aligned right, and it's more directly associated with the form (that is, it's right after the text in the last form field) so unless we have some real evidence that users are likely to be confused I'm not sure this is a problem worth reorganizing the page for. But if you guys have another proposal in mind let me know.

What does Save Changes do if the Zotero ID has changed? WIll it rebuild everything against a new Zotero library, right?

Yep, that's what I'd expect as a user. In that scenario it should be like the first time they entered an ID and synced. (So even if it fails, the previous references should be discarded, because the ID that was used to sync them was removed by the user.)

Items 1 and 4 deal with status and are more complicated. The mockup assumes a pretty simplistic approach to status notification. If we want to stick to simplistic I'd suggest that if the sync fails, we show a flash message notifying the user of the failure:

and add a line at the top of the synchronization status (before the "Exhibit items have never..." or "Exhibit items were last synchronized..." lines) that says:

Zotero synchronization failed on 2017-03-13.

I'd only show the last failure date and of course omit the text when the sync next succeeds.

In this simplistic approach, my answer to question 4 would be yes, the status line text, and maybe a flash message like:

We could also handle items 1 and 4 in a more informative but more involved way, such as doing something similar to the status progress we show for indexing items. I didn't suggest that previously because I know we spent quite a bit of time getting that implemented and assumed that might be overkill for this feature, which is likely to have relatively few consumers. However, @jkeck probably has a pretty good handle on the item indexing status approach at this point so if @blalbrit and the team think we can do something along those lines for status, let me know and I can come up with a mockup for displaying status that would be more informative than the more simplistic approach proposed above.

[jkeck]

I think the flash messages assume that either a page load has happened (e.g. redirect after form submit) or that the user hasn't navigated away from the page (which I don't think we can assume).

Perhaps we're overthinking this a bit too much, particularly as @ggeisler mentions, this is likely to have few consumers. The strategy that we take for the indexing status display would add significant complication to completing this issue.

[blalbrit]

+1 @jkeck - I don't think we need to take the most expedient approach here.

[mejackreed]

@ggeisler Is the intent here to view these as an unordered list in the UI <ul>?

Or use separated <div>'s

[ggeisler]

@mejackreed I think unordered list items, unless you see a reason why using separate divs might be better/more flexible?

[mejackreed]

@ggeisler potentially if users wanted to copy and paste the bibliography, that may present problems? I don't know.. either way seems fine

[blalbrit]

@mejackreed @ggeisler - are we still talking about output from Zotero, specifically to support Parker bibliography - or has the conversation moved into a different area?

[ggeisler]

@mejackreed Yeah, maybe. I guess I just hesitate at divs because we might have to apply some additional styling to make a long list of items as easy to parse as what we get for free from a ul (e.g., indent after first line of each entry, add extra space between each one, or add a separator between them, etc.)

I don't think it matters too much and we could adjust if we get feedback. If @blalbrit has strong feelings we could go with what he prefers.

[ggeisler]

@blalbrit Yes, we're talking about the bibliography output on the exhibit item detail page. The list of references and how they are presented, visually.

[mejackreed]

Ah.. that makes sense @ggeisler . I'm 👍 for the list.. just wanted to discuss the different avenues.

[blalbrit]

my only requirement is that the bibliography be alpha-sorted and in a standard bibliography format (preferably Chicago).