Make ingest Webform Workflow simpler and natural

[DiegoPino]

The Problem

Our webforms integration works by dynamically loading a Webform, with a special Webform Handler from inside a Widget logic. Since Every widget is also loaded inside a wrapper, Node Entity Specific form, we have Basically Two different "Save" buttons.

• One to Preserve all the data input coming from the user while filling this webform, and handled by our custom handler on the final (or sometimes the only) Webform Step
• The Content Entity Form (a class initialized by the Content Entity Class) that takes our Widget input, but also any other Input and actually triggers the Node Save action.

What we want

We want a few things here, most related to the Issue at hand, but also a bit wider into the Webform integration thing (like Ajax used, what happens if they don't want ajax?) and this leads to different tasks and solutions:
1.- We want to research use cases/empiric workflows using the current code. Like what are the normal operation steps on ingesting an Object. Are we displaying/hidding enough functionality so Users don't get confused by the fact there is "Saving Metadata" action, and a "Save the whole Node" action and both are different buttons? Other ones

• what happens when people do things we have not expected, like preview, back, edit, preview again. etc.
• How do we deal with the fact that Users can customize their Webforms a lot? Can we enforce in our widget enough defaults so even when they do change per webform settings we are still operational and no data gets lots?
• Can the user go back to edit data once it pressed the "Webform" version of Save button?
• Are our Labels explicit enough?

So this first part is building use cases, building webforms and Form modes and testing them thoroughly, annotating issues, bugs, problems and in general worklow expectations v/s reality

2.- We want a more (as an Widget setting) seamless ingest experience, which in my opinion means:

• Single Save button. Which of the two buttons is the one triggering the metadata save and the node save is of course the question, and is directly related to how good we are dealing with Drupal's Form API which is not a simple tasks. Gist here is Buttons have Submit handlers, those get attached on Form build and take over functionality. Also there are Validation callbacks that run.
  • We could make sure that one of the buttons is able to call the other button's callback if its working in the context of a Widget (means the webform is used to fill metadata, not just testing or building it).
  • We could skip validation steps or do programatic validations on existing, loaded data so people can jump from one step to another. We need for that make sure in which context validations runs.
  • We can keep temporary values longer around, or write them every time someones passes a validation. That way people could, e.g change Page, come back to a new Object and they could continue filling Metadata without having to press "Webform Save" button.
    Note: we need to have language here to speak about the two save Buttons: Lets call the Webform one "Webform Save", and the other one "Node Save". Same goes with "Preview" buttons.
• It would be great to be able to operate without Ajax too. Which would imply we "We can keep temporary" values around, and when user finishes filling the webform, it gets redirected to the node edit form again and those temp values are loaded so they can fill out the rest

3.- As part of this task, we want Webform to "inform" data to other Fields and their Form widgets, in specific The Node Label. Because the Node Label displays its own widget (breaking the natural rendering order and making some people angry), but Node Label is required and a must for an ingest (If we plainly hide it?Ingest simply dies!) we need to add code to our Webform widget and settings so any Webform Element (our own internal Webform "title" input) can feed, on Node Save its value into the Node Label. This could be even done more generically for other Node Fields too.

This is a large and super-duper important Issue. I feel we have come pretty far with this but now we need to become Webform Pros and Form API wizards. No more time to loose and all this is needed. I would say from this one at least 4-5 Pulls can come out, but also a full set of Webform Research experience and long UI/UX driven conversations. I can share in the comments some Code pointers to get this started.

@marlo-longley @mitchellkeaney @giancarlobi

[mitchellkeaney]

To get the conversation started w/r/t to the first item here: I think a change that could significantly improve the ingest workflow is what you stated last: making the functionality of the buttons more explicit to inform the user exactly what the button(s) is/are doing. Something like "Save Metadata" vs. "Save Digital Object" (the first persists all the metadata and the latter uploads the object into the repository).

[giancarlobi]

@DiegoPino Sorry in advance for this note, a little bit more general about webform and ingesting.
I was talking few minutes ago with Anna about this and my first question was: do we really need a so complex webform for ingesting? Anna answered me absolutely yes! Why?
Well, we have to consider two mainly needs: Cultural Heritage and Institutional Repositories.
The last one is the reason for “user ready” webforms because Horizon2020 requires institutions load publications onto an IR before any other private/public editors, so institution’s researchers are to be able to independently load/ingesting own documents onto Archipelago by a friendly/simple/clear webforms. Maybe we need a second step where Librarian check/edit and approve what user done before make it public/restricted available.
Regarding Cultural Heritage Preservation, which is more my focus, we have a different workflow. In this case the main actors are System Manager, Librarian and Digitization Staff. The first one manage files ingesting by batch scripts or something like IMI while Librarian load/edit metadata by electronic sheet or IMI, so for this kind of use webforms are not so important.

[marlo-longley]

@giancarlobi Thanks for these cases, it is good to remember that batch ingest means Webform use will be for other, specific cases. Your example strikes me because our most recent Archipelago deployment provided a similar workflow (on a much smaller scale!!) - user-entered webform content goes to approval by curator/Librarian. Thinking about these stages can help us. For example the Draft/Published Node Save button is strange now, appearing on multiple wizard pages.

This is a hard issue because descriptive metadata holds almost anything (good thing!), there is no inherent UI coherence about how to interact.

More soon after talks tomorrow with Diego and Mitchell.

[mitchellkeaney]

One analogy discussed was something like a "Save Progress" button when using a wizard for an ingest workflow. This allows a user to enter as much information about an object or collection, reach a "stopping" point, Save Progress, then return when they have more time, etc. to fully Save or Complete their ingest.

[mitchellkeaney]

(Moving my note here since it's directly related to making webform workflow more helpful and not so much related to the topic of issue #41 )
@DiegoPino @giancarlobi, continuing off the third "•" in this comment from issue #41

Perhaps a simple way we could make the "Descriptive Metadata" Webform more helpful / useful / intuitive would be by directly employing the Drupal Help boxes to sort of guide users through where they can go to make modifications and/or provide short explanations for more context on what is happening inside the webform. Since "Descriptive Metadata" is the demo Webform that ships default, by nature, we sort of have to have a very vanilla Webform that almost imposes certain use cases. Fortunately for this project we do have some use cases where people want to really open up the Archipelago box, ignore the vanilla webform and move things around before getting started. Though, it seems we've run into more instances of users wanting to sort of know how everything is working (in other words, "upload and go") before they start tugging on patch cables and wires.

[DiegoPino]

@mitchellkeaney my fear is we could end putting videos/help boxes for things that will be/could be customized. If people, e.g change some fields, the help boxes will still be around and not match. But! Webform has also non input elements. What if you write/bring that help into the webform itself?

[mitchellkeaney]

dang, did not even think about the help boxes staying static, haha... 🤦
Can you give me an example of non-input elements in the webform, I'm sure what you mean by that?
But ultimately, yes - I can write up these help suggestions to be added in one way or another!

[DiegoPino]

Webform (elements) has Markup elements. Those serve as ways of outputting data/design/info without expecting people to add anything. The Webform docs are quite good (best ones i have seen in Drupal https://www.drupal.org/docs/8/modules/webform/webform-features#s-elements

[mitchellkeaney]

perfect, thank you for linking this!