3090 v2 distribution confirmation

[danielabar]

Resolves #3090

Description

Introduce a Stimulus controller for the new distribution confirmation feature. It intercepts the Save form submission (only for New distributions, not Edit, as per earlier disucssion).

This controller extracts key information from the new distribution form including:

• Partner
• Storage location
• Items and Quantitites

And displays these in a modal asking the user if they're sure this is what they intend.

The user can click "No...", which will close the modal and then they're still on the new form where they can continue filling it out, making changes, etc.

Or the user can click "Yes..." in which case the form is submitted, and then the existing server-side flow runs.

Note: There's a previous PR from an earlier attempt which introduced a new pending status for the Distribution model and saved the distribution in this state as part of the confirmation flow: #4341. But after some discussion, it was decided to go with a simpler client-side JS approach as per this PR.

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?

• For manual testing, create a new distribution and click Save - the new modal should appear.
• The existing distribution system tests were maintained to expect the modal display after clicking Save from a new distribution.
• A new distribution system test was added to verify that user can click "No..." from the confirmation modal, and remain on the new view.

Screenshots

Here is an example of the new distribution confirmation modal, shown after user clicks Save:

[cielf]

I'm waiting to do any more functional reviews until the issue with bin/setup gets addressed. (Just setting expectations)

[cielf]

@danielabar Hrmm. Now that I see it in action, I want a small adjustment here -- let's change "meant" to "want".

Reason: We are displaying this before any error checking. The past tense of "meant" implies you are pretty much finished. "Want" is better for setting expectations of still being in the process. (and it still works if we can do the error checking first)

[danielabar]

Updated the wording.

[cielf]

@daniealbar Thank you very much for your hard work on this so far! (I really appreciate your thoroughness)

Alas, During the earlier discussions, I failed to catch that the confirmation would be before basic error checking and the combination of multiples, etc.

Does it have to be that way, with this approach? (I'm concerned that it won't meet the need to have useful numbers for their final review)

(also calling @dorner into the discussion)

[danielabar]

@cielf Currently the server side part of this performs validations and saves. And this being client side, it runs before all that.

Would a more ideal flow would be, from clicking Save:

1. Run validations (but do not create new distribution, even if valid)
2. If invalid, render the form with errors
3. If valid, render the form with the modal confirmation dialog
4. From clicking "yes" on modal, again submit to server for validation and creation (should be valid by this point, but just in case since its still a form coming from the client)

It might be possible to introduce a new endpoint for validation only. It may also be possible to cause some Stimulus code to trigger as a result of render (saw something like this in a tutorial, could look into this).

A tricky thing will be the validation though, I ran into this earlier when working on the first attempt pure server side. For basic validation like is partner and storage location filled in, that's straightforward. But for inventory, I it seems to be coupled with actually decrementing the inventory amount in this line in DistributionCreateService: distribution.storage_location.decrease_inventory(distribution.line_item_values).

There's also some validation about this distribution already being fulfilled (if it was created from a request) that also happens together in the creation service, making it difficult to tease apart creation from validation.

There's another complexity with a further inventory check that happens after creation success wrt on-hand minimums. Although that may be only a flash alert after the fact, but doesn't stop creation.

So this might require some server side restructuring of the code to pull out all the distribution validation logic into one place that can be called independently of saving/updating anything.

[cielf]

@danielabar
My gut says that would be better. I had hoped to do a bit of UX research today to see if I'm totally off base on this, but did not get to it.

If I'm understanding correctly, It sounds like this would be a bit of a mess, I mean challenge... g and that it that could make the code harder to understand.

The distribution confirmation is a "nice to have" , rather than core functionality. We'll have to make a judgement call on whether the benefit derived is worth the increased complexity of the code. I'd like to get @dorner 's take on this, see if there's an angle you haven't considered, but also to get his read on maintainablity.

I know you've put a lot of work into this one.

[dorner]

I think the idea of separating out validation from creation is a good one. I think it'd make it easier to understand, not harder, because we aren't shoving them all together. I think that should happen in a separate PR though. There also should be a focus on event sourcing, so maybe validating inventory items is not quite as important. (I don't think there's a direct method to say "initialize this event and check if the inventory can handle it", but it's not hard to do - maybe 2-3 lines of code.)

Once we have them separated, the flow becomes a lot easier to work with and we can more easily have that confirmation page.

The fact is that without a way to validate the distribution without creating it, this ticket is pretty much impossible to complete. Whether this is the most important thing for Daniela to spend her time on is a judgment call for @cielf I think.

[cielf]

IIUC, the stuff that we'd have to do to make this happen is a good idea anyway?

I am, however, starting a conversation with @danielabar over in slack about the partner profile rework - which is in the important but we're having trouble getting to it bucket.

[dorner]

OK - I think the action for this should be to create another issue around separating validation from creation logic, and blocking this issue on that one. Prioritization is a separate question.

[cielf]

I'm not sure I know enough about this to write it up -- though one question would be whether we want to separate it just for distributions, or across the board of itemizables? (for consistency's sake, I would think the latter)

[danielabar]

An attempt at new issue title/description:

Title: Refactor Distribution Creation to Separate Validation from Creation

Label: refactor

Description:
Currently validation and creation of a Distribution are coupled, making it difficult to insert a confirmation feature into the workflow as per issue #3090. (i.e. confirmation should only be displayed for a valid, but not yet created, distribution).

Details:
When a new distribution form is submitted, the create method of app/controllers/distributions_controller.rb is called. This method creates an instance of a Distribution model from the form parameters, but doesn't call the valid? method on this model. Rather, it calls DistributionCreateService which, among other things does:

• Validate that if this distribution is associated with a request, that the request has not already been fulfilled. This is done via a method in DistributionCreateService rather than with a validate declaration on the Distribution model.
• Calls save! on the distribution model, which at this point, will run all the declarative validations on the Distribution model. This includes an inventory check line_items_exist_in_inventory, which is defined in the Itemizable concern app/models/concerns/itemizable.rb.
• Decrease the inventory at the storage location, via the decrease_inventory method of the StorageLocation model. This method also combines updates and validation, and will raise if there are insufficient items.

What we'd like to do is separate out all the validations so that they can be called separately from creation, ideally from a single call to distribution.valid?. This way when DistributionCreateService calls distribution.save! it runs the exact same set of validations as distribution.valid?.

The benefit is this would allow for a future endpoint that only does validation via distribution.valid?, to build the confirmation feature something like described in this comment.

TODO: Are there any significant differences between Itemizable validation line_items_exist_in_inventory and the validation that's embedded in StorageLocation#decrease_inventory?

As for expanding to all the Itemizables, I haven't had a look but it's possible that the business rules and validation implementation could be different on each, so it may keep the PR more manageable to have separate tickets, starting just with Distribution model, since we know it has the issue of validation coupled with creation.

[dorner]

Let's keep it to Distribution.

The inventory thing should not be a focus of this. In event world, we're going to get rid of inventory items entirely, so I'd ignore it as much as possible.

[danielabar]

Thanks for the clarification, this simplifies things in that it's only the "is there an associated request that's already fulfilled?" validation that isn't part of the distribution validations.

Thinking about this further, if distribution.valid? already does much of the validation, it might be possible to continue on my original stimulus branch, with having it call a new validation only endpoint. I'll attempt this (i.e. may not need that refactor ticket after all).

[danielabar]

@cielf I've made some changes so that first it will run some basic validations on the distribution before showing the modal (from user clicking Save on new distribution).

For now, this runs the existing validations defined on the Distribution model via distribution.valid?. I didn't change any of that logic given earlier comment wrt inventory.

And there might still be a case of "request already fulfilled", but I wasn't sure if that validation actually belongs in distribution model (eg: that would also run on update).

Also wanted to see if this is closer to the desired flow wrt validation. Can you try this out whenever you have a chance. For example, if you leave out Partner and try to Save, it won't show the modal, instead it will show the validation error. Then if you fix the error by filling out partner and Save again, then the modal will show.

[cielf]

You're on my list! -- it's a little long at the moment (managing expectations)

[cielf]

This is definitely closer! It feels a lot smoother to me at this point.

Is it plausible to handle the case where someone has entered the same item twice before we show the modal?

[danielabar]

re: dupe items - How is this handled currently?

I tried on main branch, to create a new distribution with dupe items and it was allowed, didn't see any warning/validation error:

[cielf]

Yeah -- it is allowed. They currently get smooshed together using the distribution's method combine_distribution, which is called before_save.

("No, it' not plausible" may be a valid answer!)

[danielabar]

I had a look at Itemizable#combine!, which is called by Distribution#combine_distribution. I don't think it will be possible to use that from the JS modal as that method is building ActiveRecord models to be associated with the distribution, just before the distribution gets saved. It actually deletes the existing ones it got from the form and builds new one based on the combination, by unique item_id.

But in the confirmation modal, it's client side and only has what it can parse from the DOM. It might be possible to write a similar combine method client side in JS, that would only be used for display purposes in the modal. I can look into this.

[danielabar]

Got the combine display working in the confirmation modal, see this commit for details.

[danielabar]

Just to clarify, the combining of line items in the modal is display only, it doesn't modify the model because it hasn't gone to the server yet. So if user clicks "No I need to make changes", the line items in the form will be exactly as the user originally entered them.

[cielf]

It may be tomorrow before I have a chance to try this out, but it sounds workable to me from a functionality pov

[cielf]

Functionality looks good. Over to @dorner for technical review.

[dorner]

Had a suggestion to tweak the approach. Overall this looks like great work!

[dorner]

Do we need a special stylesheet just for one rule? :)

[danielabar]

Moved into custom.scss

[dorner]

In general it's better not to skip this unless there's a good reason to. If the issue is JavaScript, there is a way to send the CSRF token so this passes: https://bloggie.io/@kinopyo/sending-non-get-requests-with-fetch-javascript-api-in-rails

[danielabar]

This is a strange one, with the current code that extracts form data, the authenticity_token is indeed being submitted, and validated by Rails successfully, at least, the first time.

But in the scenario where the first time the form is invalid, then user submits again with same/different validation errors, the correct token is being included, but Rails says its invalid.

I tried also with the technique mentioned in that blog post (using header rather than in post body), but the same issue occurs.

Thought it might be due to JS DOM parsing or Stimulus lifecycle keeping a reference to the old token, but that's not the case, it is submitting the latest token.

Requires further investigation if the token verification needs to be there also for the validate endpoint.

[danielabar]

Did a little more digging into ActionPack (lib/action_controller/metal/request_forgery_protection.rb), where verify_authenticity_token is implemented:

It tries three possible ways to validate the given token:

compare_with_global_token(csrf_token) ||
            compare_with_real_token(csrf_token) ||
            valid_per_form_csrf_token?(csrf_token)

The very first time the form is submitted (and Stimulus controller intercepts and submits POST /distributions/validate.json, compare_with_global_token(csrf_token) returns true.

But on all subsequent attempts, compare_with_global_token(csrf_token) and compare_with_real_token(csrf_token) return false, and so it attempts the last option valid_per_form_csrf_token?(csrf_token).

That last method calculates the correct_token based on the request path and method:

correct_token = per_form_csrf_token(
            session,
            request.path.chomp("/"),
            request.request_method
          )

I put a breakpoint here and compared correct_token when the validate endpoint is submitted vs the create endpoint. Since the request.path is different, /distributions.validate.json vs /distributions, it doesn't match for the validate endpoint.

So I think the token that gets generated in the form upon a render from the first validation error, represents a token for the create endpoint, and so it won't be correct if the validate endpoint is submitted.

[dorner]

Remove this line?

[danielabar]

Removed

[dorner]

Do we need this in production?

[danielabar]

Removed

[dorner]

I'm not the biggest fan of basically reinventing ERB with this. 😂 The combination of line items is also something that we already have implemented in Ruby and have to do again here.

I'm wondering if we can reuse just the ERB for this. Something like:

• Send the fetch for validation
• If valid, it sends the HTML to render, e.g. {"valid": true, "body": "<div ...></div>"}
• Otherwise you submit the form as usual

You can build the distribution and line items and render the confirmation form with ERB that way. Something like

def validate
  @dist = Distribution.new(distribution_params.merge(organization: current_organization))
  @dist.line_items.combine!
  if @dist.valid?
    body = render_to_string
    render json: {valid: true, body: body}
   else
    render json: {valid: false}
   end 
end

Then you just add validate.html.erb and use it like you would any other view. The Stimulus controller becomes super simple in that case, you just do something like modalTarget.innerHTML = response.body.

[danielabar]

Implemented, a few notes:

Had to specify the format as html (otherwise was trying to render json) and layout false, to get a focused view of only the modal body:

body = render_to_string(template: 'distributions/validate', formats: [:html], layout: false)

Indeed this greatly simplified the stimulus controller, I was able to remove most of the targets. This could make it easier to re-use for other model confirmations in the future, by passing in a different preCheckPath value.

[dorner]

Can we hardcode the names? Faker has bitten us with this before. :)

[danielabar]

In the case of spec/system/distribution_system_spec.rb, the storage location and partner names are defined with let and FactoryBot near the top of the file:

let(:storage_location) { create(:storage_location, organization: organization) }
let!(:partner) { create(:partner, organization: organization) }

Then used throughout, for example:

select partner.name, from: "Partner"
select storage_location.name, from: "From storage location"

Did you mean to change the FactoryBot create to pass in specific names and then replace throughout the entire spec file? Something like:

let(:storage_location) { create(:storage_location, organization: organization, name: "Test Storage Location") }
let!(:partner) { create(:partner, organization: organization, name: "Test Partner") }
...

# Later in tests

select "Test Partner", from: "Partner"
select "Test Storage Location", from: "From storage location"
...
expect(find(:element, "data-testid": "distribution-confirmation-partner")).to have_text("Test Partner")

And so on for all occurrences?

btw, the storage location factory already has a hard-coded name:

name { "Smithsonian Conservation Center" }

Although using that in the distribution system spec expectations might create some confusion as to where that value from.

The partner factory on the other hand uses some variability, although with a sequence rather than faker:

sequence(:name) { |n| "Leslie Sue, the #{n}" }

[dorner]

Did you mean to change the FactoryBot create to pass in specific names and then replace throughout the entire spec file?

Yep, exactly! The factory values are IMO something that should only be created but never referenced, because as you say it's really confusing as to where the values come from. If we need to check the value, we should always be providing it ourselves.

[danielabar]

Updated to specific partner and storage names.