[api provider - collections] UX Design - API Provider Collection Support

[dongniwang]

This is a...

[x] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[ ] Bug report  
[ ] Documentation issue or request

Description

Per request in #4291, opening this issue to track UX design for API Provider Collection Support.

The design will reflect:

• Updates to the connection configuration page - adding an option to treat the produced data as a collection
• Changes to step list - adding "For Each" step
• Changes to IVP once the foreach step is added

Also need help and suggestion from @TovaCohen regarding the desired wording: open/closed foreach vs. open/end foreach.

cc: @lburgazzoli @christophd @amysueg @gaughan

[lburgazzoli]

@dongniwang @gaughan

I was thinking if it really make sense to ad an option to treat the produced data as a collection or if it could be better if the back-end translates old style integration to the new style so:

• the back-end will support old style integrations so if an old integration is started without modifications, it will work as today
• the back-end translates the implicit split to the foreac/endforach when serving the data to the UI (similar to what happens to integration not using flows) so when the user saves the integration, such integration gets automatically upgraded

This has the advantage that it make components more consistent as the data shape is defined so they will have a single way of working and is - imho - less confusing for the users.

What do you think ?

@christophd @zregvart @heiko-braun

[christophd]

If we intend to remove the implicit split capabilities completely from Syndesis in future I think the automatic translation is a good option as the user then will not see any split yes/no options on the connector and we should hide those because at some time we will remove that implicit split completely.

In case we would like to keep the implicit split capability for some reason I think the option split yes/no on the connector is a good way to go.

If we do the automatic translation to new style integrations the foreach steps just pop up all of a sudden when users open/edit old integrations. Compared to that an additional split yes/no option that is introduced on connectors is less disturbing I would say.

I think we should decide based on keeping the implicit split functionality or not.

[lburgazzoli]

On Thu, 24 Jan 2019 at 11:01, Christoph Deppisch ***@***.***> wrote: If we intend to remove the implicit split capabilities completely from Syndesis in future I think the automatic translation is a good option as the user then will not see any split yes/no options on the connector and we should hide those because at some time we will remove that implicit split completely.

Yes the implict step should go away

I think the option split yes/no on the connector is a good way to go. If we do the automatic translation to new style integrations the foreach steps just pop up all of a sudden when users open/edit old integrations. Compared to that an additional split yes/no option that is introduced on connectors is less disturbing I would say. I think we should decide based on keeping the implicit split functionality or not.

I do agree that the split yes/no flag may be less invasive but it will cause confusion to the iser as there will be two way of doing the same thing. For us it introduces some additional cases as the connector data shape then need to be adapted so yet another if/then/else and troubles when upgrading connectors. Let put this on PM and UX :) —

You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByEhfuDXD7K3OkIF6DQ7swTueGSK3eVks5vGYSPgaJpZM4aIJkc> .

-- -- Luca Burgazzoli

[gaughan]

:) nice offload Luca!
I agree with the approach, it will be a new experience to have a step automatically be added but hands the user control now.
Moving datashape manipulation to a step seems more natural than a connection setting too.

[zregvart]

Can we do this in the (database) migrations?

[lburgazzoli]

It is not needed but can be done there too (if really needed).

[zregvart]

I kinda think that migrations, being a one-off thing, would reduce the tech debt. Though I would not insist on it, could be difficult to write/test, it is JavaScript... (sarcasm, right)

[lburgazzoli]

Yes that’s possible, however the problem is that we should also handle the case of when someone import an old integration or when someone move things between envs. So I’m not sure about what’s the best trade-off

On Thu, 24 Jan 2019 at 21:40, Zoran Regvart ***@***.***> wrote: I kinda think that migrations, being a one-off thing, would reduce the tech debt. Though I would not insist on it, could be difficult to write/test, it is JavaScript... (sarcasm, right) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByEhVp97wjt06ZPqsU8G0HxgFuhyl7Wks5vGhoigaJpZM4aIJkc> .

-- -- Luca Burgazzoli

[zregvart]

Already done:

syndesis/app/server/endpoint/src/main/java/io/syndesis/server/endpoint/v1/handler/integration/support/IntegrationSupportHandler.java

Line 349 in 8855b59

migrator.migrate(given, version);

[lburgazzoli]

ah great, so let me open an issue for that, thx

[lburgazzoli]

@dongniwang @gaughan

Additional food for thought for UX / PM

Assuming we have a simple integration that just logs records from the db:

If we add forEach/endForEach block as a single "step" then the integration should become:

Which means that the log step is not more the final one, which introduces a new behavior as then one should be able to add a step after the endForEach

Things are going to be different for an API Provider

As in this case the final step is fixed

[gaughan]

👍
@dongniwang Look good to you?

[christophd]

At the moment we can not change the order of steps once they have been added. Also we can not place new steps after the finishing step.

What does this mean in terms of how to add a foreach block? I see two options:

Option 1:

Treat foreach as a single step that adds both start and end with one single user action. The user then is able to add new steps to the foreach (= placing new steps between start and end)

Option2:

See start and end as separate steps that need to be added separately to the integration. So user can add a foreach start step at any place in the integration and then needs to add a foreach end somewhere else. Maybe also leave out the end step so the rest of the integration is automatically inside the foreach block

[christophd]

In my demo I used option 2 where I added foreach start and stop as separate steps. So this would imply less changes in the UI I guess at it is working with what we have right now. Only thing we might need to add is the ability to add the endForeach step as a finishing step after an existing finishing step.

I think at the moment only connector steps can be placed as a finishing step and this has to be done at the very beginning of designing an integration. Basically we have to first specify the starting and finishing steps when creating an integration.

[gashcrumb]

I think having another + after the last step and having a step instead of a connection would would be doable with #4322 implemented.

I wonder, do we technically really even need a last step/connection? We force it now but I don't recall the reasoning other than ensuring that an integration actually does something I guess.

[gashcrumb]

Ah, I think it was because of the data mapper, i.e. you can't really add a data mapper step until you have 2 things to map between. But, we could just hide the data mapper step in that case easily enough.

[heiko-braun]

I think option 1 is more comprehensible – treat adding start and end as an atomic operation (in terms of the interaction model). Same for removal. If we keep them separate, it might lead to inconsistent flows.

[gashcrumb]

Only problem I could foresee with option 1 is allowing the last step to be inside the foreach where it then operates on single items vs the collection, unless we also stop forcing the user to pick an end step.

[zregvart]

I'd go with option 2, and with different visualisation than other steps. Perhaps something looking like a bracket encompassing steps within.

[christophd]

@gashcrumb technically we do not need the end step (we did not have it before with the implicit split) but if we have it we are much more flexible in what we can do in an integration and we definitely need it for all those use cases where we need to do some things after a foreach

[christophd]

Adding the endForeach step to an integration also has impact on follow-up data-mapping steps as these should not see the output messages of foreach internal steps but only the overall foreach out message. see #4355

[heiko-braun]

Adding the endForeach step to an integration also has impact on follow-up data-mapping steps as these should not see the output messages of foreach internal steps but only the overall foreach out message. see #4355

Does that explain why we should use option 1 or 2?

[christophd]

Does that explain why we should use option 1 or 2?

Not really - it just points out that we need the endForeach

[christophd]

I think option 1 & 2 totally influence the way how people are writing integrations. With option 1 you will have to add foreach first and then afterwards adding more steps into that. With Option 2 you can do that same thing plus add a foreach around some existing steps that are already there.

[gaughan]

I like option 2. We should rrename here to split and merge also. Spilt could later be expanded to add more functionality like spilt the first 3 records etc.

[dongniwang]

Just wanted to capture the discussion with @gaughan and @amysueg - it seems like we can go ahead with two steps: a split step, and a merge step, given the fact that in the future we'll want a fully functional split step. And to support the current use case of api provider collection support, we are supplying users with a split step with limited functionality. Does this make sense?

Also to note, it's a UX challenge to remind users to add the merge step once they added a split step when creating an API Provider integration...but we can certainly explore a solution for this challenge.

[gashcrumb]

@gaughan @dongniwang Please consider join instead of merge, I think it's a more accurate word in this context really.

[amysueg]

@gashcrumb as @gaughan pointed out "Join" has SQL connotations... that is why we are recommending merge, a little more neutral.

[lburgazzoli]

Just want to add that merge/join/aggregate here in not generic and it is linked to the split step, in other words, at the monent, split can live alone but merge can’t.

On Fri, 25 Jan 2019 at 21:28, Amy Glass ***@***.***> wrote: @gashcrumb <https://github.com/gashcrumb> as @gaughan <https://github.com/gaughan> pointed out "Join" has SQL connotations... that is why we are recommending merge, a little more neutral. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AByEhRK7A82t74sEX9BfWyvs8o0BLM-Xks5vG2kLgaJpZM4aIJkc> .

-- -- Luca Burgazzoli

[christophd]

Why not using the integration pattern name "aggregate". As the aggregate pattern per definition can also live alone in an integration we might enhance that step at some time in future to also work without a previous split.

For instance there could be an integration collecting a fixed number of events from an event stream and then fire up a new event or do something else when the aggregate is completed.

We could reuse the aggregate step that we introduce right now for this in future.

[TovaCohen]

Are "Iteration" and "Collection" worth considering for the step names? The current step names are nouns. The descriptions have the verbs. In context, this would look like:

Advanced Filter Continue the integration only if criteria you define in scripting language expressions are met.
Basic Filter Continue the integration only if criteria you specify in simple input fields are met.
Collection Combine integration data and execute subsequent integration steps once.
Data Mapper Map fields from the input type to the output type.
Iteration Split integration data and execute subsequent integration steps once for each result.
Log Send a message to the integration's log.
Template Upload or create a Freemarker, Mustache or Velocity template to define consistent output data.

[gaughan]

@gashcrumb using join to me, may lead the user to think they are performing a SQL join. wdyt? Future DV functionality will only add to this.
@christophd +1 on aggregate, especially on expanding in the future.
@TovaCohen I think the name has to be strongest as user tend to gloss over the description, returning to read in detail only when they encounter unexpected behavior.

[lburgazzoli]

@dongniwang what is the status for this ?

[dongniwang]

The design is ready for review. Please provide your comment in InVision. Let me know if you need to set up a call to walk through the design. Thanks! https://redhat.invisionapp.com/share/JNQ9E1A4AVQ

@TovaCohen - Please review the wording and provide your feedback. Thank you!

cc: @lburgazzoli @christophd @gaughan

[TovaCohen]

I added comments to the design on pages 3, 8, 11, 12, 16, 24. Just suggesting text changes.

[lburgazzoli]

@paoloantinori @gashcrumb please have a look

[gashcrumb]

I had a look, I think it's fine. I suspect some elements of the design may not make the initial effort (the bits resembling the guided tour mostly) but it gives us a direction.

[dongniwang]

I had a look, I think it's fine. I suspect some elements of the design may not make the initial effort (the bits resembling the guided tour mostly) but it gives us a direction.

@gashcrumb - that's fine. Alternatively, we can just use the modal dialog and have the modal on the homepage when users first log on to the app after the release.

[amysueg]

The design for API Provider collection support (split/aggregate) is complete, see design tracker: https://syndesisio.github.io/syndesis-ux/

Closing this issue.

@christophd Please feel free to open additional UX issues.