Apply custom validation to filters

[SilvanCodes]

Is your feature request related to a problem? Please describe.
I would love to replace some custom code with Flop and Flop Phoenix but:
I need the ability to add custom validations to filters and across filters, i.e. validation functions may depend on the values of multiple filters.

Describe the solution you'd like
A key like

custom_filter_validations: [
  my_validation_one: {
    # field names the function needs as input
    [:filter_field_one, :filter_field_two],
    # the actual function to evaluate
    fn filter_field_one_value, filter_field_two_value -> case is_okay() do
      true -> :valid
      false -> [some_key: "my validation error message"]
    end
  }
]

that one may add to the opts argument of Flop.validate or Flop.validate_and_run.

Alternatively every validation function could also just always be called with a map of

%{filter_field_one: filter_field_one_value, filter_field_two: filter_field_two_value}

and one matches on the wanted values.

Describe alternatives you've considered
I could not use Flop. :(

Additional context
I'm willing to sketch a PR out if you agree that something along those lines would be a nice addition.

[woylie]

I don't feel like this should be part of the library. Could you solve your problem similar to this?

  def list_pets(%{} = args) do
    with {:ok, flop} <- Flop.validate(args, for: Pet),
         {:ok, flop} <- custom_validation(flop) do
      Flop.run(Pet, flop, for: Pet)
    end
  end

  defp custom_validation(%Flop{} = flop) do
    # retrieve filter values

    if whatever do
      {:ok, flop}
    else
      # build meta struct with error
     {:error, meta_with_error}
    end
  end

#253 is about adding some functions for retrieving and manipulating filters in a Flop struct, which should make this scenario fairly easy to implement. I could also imagine an add_error function similar to the one in Ecto.Changeset.

[SilvanCodes]

Agreed, if there would be an ergonomic way to retrieve the values and add custom errors to the flop, I'd be fine doing it like that. How would Flop.run behave if it receives a flop with errors?
Would it just built the Flop.Meta carrying those errors? That'd be necessary to provide the user with feedback in the UI, I think.

[woylie]

Agreed, if there would be an ergonomic way to retrieve the values and add custom errors to the flop, I'd be fine doing it like that. How would Flop.run behave if it receives a flop with errors? Would it just built the Flop.Meta carrying those errors? That'd be necessary to provide the user with feedback in the UI, I think.

Flop.run should not receive a Flop with errors, you need to use a with as in the example. For the error response, you need to return {:error, Meta.t}, with errors, params, and schema set. You can see how Flop currently handles the error case during validation here: https://github.com/woylie/flop/blob/0.18.4/lib/flop.ex#L1333. That is the format that is expected when you pass the meta struct to one of the form functions in Phoenix.HTML.Form or Phoenix.Component.

I need to think about the ergonomics of the add_error function a bit. Could be a single function that can take either a Flop struct or and Meta struct and returns a Meta struct (but the original parameter map and options are needed to build the Meta struct for the error struct), or one function to build the error Meta struct and one to add an error to a Meta struct; or maybe we could have something like a validate_with function that works similarly to Ecto.Changeset.validate_change/3. I'll do some exploratory coding.

[woylie]

I've added a couple of functions to find filters for a field.

If you have a list function like this:

  def list_pets(%{} = args) do
    opts = [for: Pet]

    with {:ok, flop} <- Flop.validate(args, opts),
         {:ok, flop} <- custom_validation(flop, params, opts) do
      Flop.run(Pet, flop, for: Pet)
    end
  end

You could implement the custom validation function similar to this:

  defp custom_validation(%Flop{} = flop, %{} = args, opts) do
    %{value: one} = Flop.Filter.get(flop.filters, :field_one)
    %{value: two} = Flop.Filter.get(flop.filters, :field_two)

    if one > two do
      {:ok, flop}
    else
      meta = %Meta{
        errors: [filters: ["is invalid"]],
        # convert_params/1 would have to be made public
        params: convert_params(params),
        schema: opts[:for]
      }

      {:error, meta}
    end
  end

In this version, the errors need to be set as a keyword list.

Alternatively, I could imagine a validate_with/3 function with this type specification:

@spec validate_with(Flop.t() | map, function, [option()]) :: {:ok, Flop.t()} | {:error, Meta.t()}

Basically the same as validate/2, but with an additional argument to pass a custom validation function.

Now your list function would look like this:

  def list_pets(%{} = args) do
    with {:ok, flop} <- Flop.validate_with(args, &custom_validation/1, for: Pet) do
      Flop.run(Pet, flop, for: Pet)
    end
  end

The custom validation function needs to take and return an Ecto changeset in this case:

  defp custom_validation(%Ecto.Changeset{} = changeset) do
    filters = Ecto.Changeset.fetch_field!(changeset, :filters)
    %{value: one} = Flop.Filter.get(filters, :field_one)
    %{value: two} = Flop.Filter.get(filters, :field_two)

    if one > two do
      changeset
    else
      Ecto.Changeset.add_error(changeset, :filters, "is invalid")
    end
  end

This has the advantage that a) you don't need to know how to build a Meta struct with errors and can just use changeset functions for validations, and b) the default and custom validation is running in one step, so you'll see all the errors at once, and not _eitherv the default validation errors or the custom errors.

In both examples, the validation error is just added to the filters field instead of the individual filters. Adding the errors on a specific filter field is a bit trickier, since the filters are a nested list, and the errors must be a list of lists in that case. I don't think Ecto has a way of adding an error to a nested list element. And I'm not sure how a custom add_error function for this case in the first example should behave. There can be multiple filters for the same field, after all. An alternative might be an additional custom_filter_validator option that allows you to pass a function that just gets called as the last step of the default filter changeset: Flop.validate_with(args, &custom_validation/1, &custom_filter_validation/1, for: Pet)

[woylie]

I added the function Flop.Meta.with_errors/3 in #279, which allows you to build error responses more easily. That in combination with the new functions in the Flop.Filter module should be enough to cover your use case. There's an example in the function documentation: https://github.com/woylie/flop/blob/main/lib/flop/meta.ex#L86.

[SilvanCodes]

Very cool, I'm eager to try it out!
Thank you so much for the quick response. <3