feat(portal): Filtering, Fulltext Search, Pagination, Preloads #3751

AndrewDryga · 2024-02-23T21:13:31Z

On the domain side this PR extends Domain.Repo with filtering, pagination, and ordering, along with some convention changes are removing the code that is not needed since we have the filtering now. This required to touch pretty much all contexts and code, but I went through all public functions and added missing tests to make sure nothing will be broken.

On the web side I've introduced a <.live_table /> which is as close as possible to being a drop-in replacement for the regular <.table /> (but requires to structure the LiveView module differently due to assigns anyways). I've updated all the listing tables to use it.

vercel · 2024-02-23T21:13:37Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
firezone	⬜️ Ignored (Inspect)	Visit Preview		Mar 16, 2024 7:12pm

github-actions · 2024-02-23T21:15:25Z

Terraform Cloud Plan Output

Plan: 9 to add, 8 to change, 9 to destroy.

Terraform Cloud Plan

AndrewDryga · 2024-03-08T19:17:07Z

elixir/apps/api/lib/api/client/channel.ex

           gateway_groups = Enum.map(gateways, & &1.group),
           {relay_hosting_type, relay_connection_type} = Gateways.relay_strategy(gateway_groups),
           {:ok, [_ | _] = relays} <-
-             Relays.list_connected_relays_for_resource(resource, relay_hosting_type) do
+             Relays.all_connected_relays_for_resource(resource, relay_hosting_type) do


Since we have various kinds of returns I've decided to come up with a convention just for us:

list_* returns {:ok, results, metadata} - basically a paginated query used everywhere in the UI

all_* returns {:ok, results} - used in places where we need to really fetch all the results without pagination (eg. list of enabled providers when we build the sign in form)

all_*! returns results or raises an error. This is when we just do the query and there is no need to wrap it in the tuple, mostly used for internal stuff like background jobs

AndrewDryga · 2024-03-08T19:19:40Z

elixir/apps/domain/lib/domain/accounts/account/query.ex

    )
  end

-  def by_slug(queryable \\ not_deleted(), slug) do


I've removed \\ queryable() everywhere so that our domain modules are more explicit, before you had to go to query and see if it will return all, or deleted, or active, or whatever records. Now you have to explicitly set that as a first stage when a query is built.

jamilbk

Not going to do an in-depth review of the code (will leave that to @bmanifold after he's back).

Some usability fixes / requests:

I don't think the Showing 25 of 304 is updated when flipping through pages (e.g. on the Groups page). Is that intended?
Clients page: Could we add sorting to the User column too? I think that will be used quite a bit. And maybe add the "All / Online / Offline" button group (or allow sorting by status).
Policies -- add sort for Group and Resource
Can you space the sort form elements horizontally to make better use of the space? e.g. this is fine on small screens (grid-cols-2 that collapses to grid-cols-1)

But on larger screens I would expect the filter elements to be side-by-side:

On the Actors page it's a little more problematic:

Maybe all of these can be contained inside a dropdown menu that's activated by a filter button?

jamilbk · 2024-03-15T17:55:19Z

elixir/apps/api/lib/api/client/channel.ex

           gateway_groups = Enum.map(gateways, & &1.group),
           {relay_hosting_type, relay_connection_type} = Gateways.relay_strategy(gateway_groups),
           {:ok, [_ | _] = relays} <-
-             Relays.list_connected_relays_for_resource(resource, relay_hosting_type) do
+             Relays.all_connected_relays_for_resource(resource, relay_hosting_type) do


AndrewDryga · 2024-03-15T18:42:49Z

@jamilbk

I don't think the Showing 25 of 304 is updated when flipping through pages (e.g. on the Groups page). Is that intended?

Yes, 25 is the page size, and 304 is the total count. We can't and don't say on which page you are currently since it's cursor-based.

Clients page: Could we add sorting to the User column too? I think that will be used quite a bit.

We can but:

I think sorting by actor name doesn't make a ton of sense because you can go to the actor and see all its clients there;
Sorting on a result of a join expression will be expensive and we won't ever be able to build an index for it so better we don't add this feature without a customer request.

Policies -- add sort for Group and Resource

Same reason here, it's not the best idea to paginate using the result of a join, you can always use a filter to find the ones you need or use a different view to see group policies (group show view) or resource policies (resource show view).

And maybe add the "All / Online / Offline" button group (or allow sorting by status).

This is hard to do because presence is not stored in the database and we need to come up with a smart way for those filters to work without hitting our database with a gigantic WHERE IN ... for large customers, so I postponed this kind of filter for the future.

Maybe all of these can be contained inside a dropdown menu that's activated by a filter button?

I'd suggest improving UX in a separate PR, this will spawn all sorts of look&feel discussions that will take time, while I think it's better if we get this merged (it's already functional and not so horrible) to unblock the clients that have thousands of groups first. There are lots of ways how filters can be improved: we hide them below the collapse button, use grids, add/remove some of the filters, etc.

Can you space the sort form elements horizontally to make better use of the space? e.g. this is fine on small screens (grid-cols-2 that collapses to grid-cols-1)

I think we need a better breakpoint so it extends on smaller screens, I'll take care of that.

jamilbk

Approving to fix the timeouts currently impacting customers. Fixes can be addressed in followup PRs.

github-actions · 2024-03-16T15:32:43Z

Performance Test Results

TCP

Test Name	Received/s	Sent/s	Retransmits
direct-tcp-client2server	231.3 MiB (+1%)	233.2 MiB (+2%)	284 (+21%)
direct-tcp-server2client	238.6 MiB (+0%)	240.4 MiB (+0%)	256 (+17%)
relayed-tcp-client2server	147.2 MiB (-1%)	147.7 MiB (-1%)	176 (-5%)
relayed-tcp-server2client	160.5 MiB (+8%)	161.0 MiB (+7%)	166 (+3%)

UDP

Test Name	Total/s	Jitter	Lost
direct-udp-client2server	50.0 MiB (-0%)	0.04ms (+21%)	0.00% (NaN%)
direct-udp-server2client	50.0 MiB (-0%)	0.01ms (+12%)	0.00% (NaN%)
relayed-udp-client2server	50.0 MiB (-0%)	0.09ms (-4%)	0.00% (NaN%)
relayed-udp-server2client	50.0 MiB (-0%)	0.05ms (+11%)	0.00% (NaN%)

…erage

…om name generator

all_*! now returns list all_* returns {:ok, results} | {:error, reason} list_* returns {:ok, results, paging_metadata} | {:error, reason}

…dering

jamilbk · 2024-03-17T12:34:57Z

@bmanifold This was merged to fix production for a customer with lots of accounts and groups, but could still use your review (and a little bit of UI/UX CSS cleanup noted in my review).

@AndrewDryga I think in the future what might be good for large PRs like this is to maybe split the PR into a UI PR and backend PR? That way we can discuss UI/UX (maybe sooner) without it impeding the backend fixes needed. Or use something quicker like wireframes to figure out how it will work/look before implementing.

AndrewDryga · 2024-03-17T21:50:23Z

I think in the future what might be good for large PRs like this is to maybe split the PR into a UI PR and backend PR?

I'm not a big fan of having dead code in the main that we can deploy at any time (like the back-end that can not be used anywhere). We can split work into smaller PRs and then merge them into a temporary branch, but this will just add more work and still require one final "big" review.

That way we can discuss UI/UX (maybe sooner) without it impeding the backend fixes needed. Or use something quicker like wireframes to figure out how it will work/look before implementing.

It's easier for anyone on our team to go and update the HTML than sit and discuss prototypes without being able to use the portal. I think with designs the best approach is to do "good enough and polish after" that rather than trying to wireframe everything and kill our productivity.

For example, I would be always faster writing HTML and LiveView code than using wireframing tools, and by doing so I can also understand the limitations better, what is easy to do and what is not, which components are available in FlowByte, while dragging square boxes around we will make decisions that then can be really hard to implement, which will require another set of discussions, etc.

AndrewDryga changed the title ~~Filtering, Pagination, Preloads~~ Filtering, Fulltext Search, Pagination, Preloads Feb 23, 2024

AndrewDryga force-pushed the andrew/filtering-and-paging branch 2 times, most recently from 4915e56 to 3af1033 Compare February 28, 2024 16:51

jamilbk mentioned this pull request Mar 3, 2024

Allow deleting synced users/groups #2708

Closed

AndrewDryga mentioned this pull request Mar 5, 2024

Update Group label links to be context-aware #3962

Closed

AndrewDryga force-pushed the andrew/filtering-and-paging branch from a53ebfa to a121c62 Compare March 8, 2024 19:14

AndrewDryga requested a review from bmanifold March 8, 2024 19:14

AndrewDryga commented Mar 8, 2024

View reviewed changes

AndrewDryga requested a review from jamilbk March 14, 2024 17:07

AndrewDryga force-pushed the andrew/filtering-and-paging branch 3 times, most recently from 6df1ec9 to ebf1f1e Compare March 14, 2024 17:48

AndrewDryga changed the title ~~Filtering, Fulltext Search, Pagination, Preloads~~ feat(portal): Filtering, Fulltext Search, Pagination, Preloads Mar 14, 2024

AndrewDryga marked this pull request as ready for review March 14, 2024 17:58

jamilbk reviewed Mar 15, 2024

View reviewed changes

jamilbk self-requested a review March 15, 2024 19:38

jamilbk approved these changes Mar 15, 2024

View reviewed changes

AndrewDryga force-pushed the andrew/filtering-and-paging branch from 2de6200 to 06f5181 Compare March 16, 2024 15:01

AndrewDryga force-pushed the andrew/filtering-and-paging branch from 5ce3b7a to 47820b9 Compare March 16, 2024 17:59

AndrewDryga added 6 commits March 16, 2024 12:45

Added keyset based pagination helpers for list and improved tests cov…

707c89a

…erage

Integrate paging and preloads into domain app

3d5c66b

Implement custom preloading

2f7c3fe

Bring accounts context up to date

05e5b3e

Add missing accounts test

ad3af4c

Add repo docs and typespecs

6eb49cb

AndrewDryga added 23 commits March 16, 2024 12:45

Bing billing up to date

5c414af

Add missing clients tests and broadcasts

b92ea95

Bring flows up to date

edeb8bc

Bring gateways up to date

54bc8cf

Use chatgpt to clean up inappropriate for a security product words fr…

aacdcd3

…om name generator

Bring Policies up to date

bcccb9f

Bring Relays up to date

883d28c

Bring resources up to date

fcf6804

Move Domain.Validator to Domain.Repo.Changeset and remove unused code

68d9434

Fix some naming conventions

469cc19

all_*! now returns list all_* returns {:ok, results} | {:error, reason} list_* returns {:ok, results, paging_metadata} | {:error, reason}

Implement filters

a2d21de

Implement custom ordering

0d0a0d7

Bump all deps

185c605

Make all index tables live

e05a93f

Push some other fixes to the domain context

7e77026

Add missing typespecs

a8a6242

Introduce live tables to the web app to handle paging, filters and or…

5bbe78d

…dering

Fix dialyzer errors

59251b6

Bump all deps

d3c714c

Fix ChatGPT typos

07f0654

Fix code style issues

aa48592

Fix credo issues

b1f350d

Resolve issue with Hex archive of Floki dependency and bump all deps

2247704

AndrewDryga force-pushed the andrew/filtering-and-paging branch from a5286f6 to 2247704 Compare March 16, 2024 18:51

AndrewDryga added 2 commits March 16, 2024 13:03

Checkout deps again when trying to migrate DB to main branch state

88bb29d

Use finch as swoosh API client

c06ef4b

AndrewDryga merged commit f3c8c73 into main Mar 16, 2024
137 checks passed

AndrewDryga deleted the andrew/filtering-and-paging branch March 16, 2024 19:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(portal): Filtering, Fulltext Search, Pagination, Preloads #3751

feat(portal): Filtering, Fulltext Search, Pagination, Preloads #3751

AndrewDryga commented Feb 23, 2024 •

edited

Loading

vercel bot commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 23, 2024 •

edited

Loading

AndrewDryga Mar 8, 2024

jamilbk Mar 15, 2024

AndrewDryga Mar 8, 2024

jamilbk left a comment

jamilbk Mar 15, 2024

AndrewDryga commented Mar 15, 2024 •

edited

Loading

jamilbk left a comment

github-actions bot commented Mar 16, 2024 •

edited

Loading

jamilbk commented Mar 17, 2024 •

edited

Loading

AndrewDryga commented Mar 17, 2024 •

edited

Loading

feat(portal): Filtering, Fulltext Search, Pagination, Preloads #3751

feat(portal): Filtering, Fulltext Search, Pagination, Preloads #3751

Conversation

AndrewDryga commented Feb 23, 2024 • edited Loading

vercel bot commented Feb 23, 2024 • edited Loading

github-actions bot commented Feb 23, 2024 • edited Loading

Terraform Cloud Plan Output

AndrewDryga Mar 8, 2024

Choose a reason for hiding this comment

jamilbk Mar 15, 2024

Choose a reason for hiding this comment

AndrewDryga Mar 8, 2024

Choose a reason for hiding this comment

jamilbk left a comment

Choose a reason for hiding this comment

jamilbk Mar 15, 2024

Choose a reason for hiding this comment

AndrewDryga commented Mar 15, 2024 • edited Loading

jamilbk left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 16, 2024 • edited Loading

Performance Test Results

TCP

UDP

jamilbk commented Mar 17, 2024 • edited Loading

AndrewDryga commented Mar 17, 2024 • edited Loading

AndrewDryga commented Feb 23, 2024 •

edited

Loading

vercel bot commented Feb 23, 2024 •

edited

Loading

github-actions bot commented Feb 23, 2024 •

edited

Loading

AndrewDryga commented Mar 15, 2024 •

edited

Loading

github-actions bot commented Mar 16, 2024 •

edited

Loading

jamilbk commented Mar 17, 2024 •

edited

Loading

AndrewDryga commented Mar 17, 2024 •

edited

Loading