Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issue on wag gui #84

Closed
bluecraank opened this issue Jan 3, 2024 · 9 comments
Closed

Performance issue on wag gui #84

bluecraank opened this issue Jan 3, 2024 · 9 comments
Assignees
Labels
bug Something isn't working needs testing (on unstable) If the feature exists, but isnt yet released

Comments

@bluecraank
Copy link
Contributor

bluecraank commented Jan 3, 2024

Here i'm back again,

I don't want to badmouth WAG - but is there any possibility to boost GUI performance?

We are probably the exception when it comes to the number of devices and users

Devices: 163
Users: ~ 155

image

Changing something on groups takes about 4-10 Seconds until its saved.

image

Maybe any idea or solution for this?

FYI
WAG is very nice!

@NHAS
Copy link
Owner

NHAS commented Jan 3, 2024

Howdy!

Unfortunately for some reason changing the ebpf map takes quite a long time. I've tried to do it in a way that changes only individual entries which should be faster, however it's actually slower then this implementation.

Could you give me your kernel and distribution please?

I can't really promise any improvements as it currently works in under a second for our currently business (<50 users)

@NHAS
Copy link
Owner

NHAS commented Jan 7, 2024

If you could give a representative sample of your configuration for rules that would be super useful!

@NHAS NHAS added the bug Something isn't working label Jan 7, 2024
@NHAS NHAS self-assigned this Jan 7, 2024
@NHAS NHAS added the needs testing (on unstable) If the feature exists, but isnt yet released label Jan 9, 2024
@NHAS
Copy link
Owner

NHAS commented Jan 9, 2024

Alright, so good news and bad news.

Good news, I've made this about 50% faster on my 177 device test deployment.

Bad news, the remaining 50% is due to the kernel syscall forcing global syncronisation before giving the EBPF program the new map containing the policies. Which was introduced recently, so it blocks for quite a while even when batching calls.
(cilium/ebpf#1297)

I've made quite a few changes that now live on unstable. So I'd be keen for you to try them out.

@NHAS
Copy link
Owner

NHAS commented Jan 9, 2024

Actually I spoke a little soon. I've done an optimization which caches some maps, and now I've gotten it down to ~30ms for 177 users.

This is on unstable and you should definitely see if it fixes your problem.

@bluecraank
Copy link
Contributor Author

Seems to work well!

image

Editing groups just takes about 16ms atm

Thank you!

@bluecraank
Copy link
Contributor Author

bluecraank commented Jan 12, 2024

Oh!

I guess its to late but:

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
Kernel 6.1.0-13-amd64

sample of our config:

"Acls": {
        "Groups": {
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:ity": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:itz": [
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:rdp-clients": [
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr"
            ],
            "group:replaced": [
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr",
                "fake.usr"
            ]
        },
        "Policies": {
            "*": {
                "Allow": [
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 10092/tcp 445/tcp 137/udp icmp",
                    "fake.domain.local",
                    "192.192.192.2/24"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 445/tcp"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 80/tcp 443/tcp"
                ]
            },
            "group:replaced": {
                "Allow": [
                    "fake.domain.local"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 445/tcp 137/any",
                    "fake.domain.local 3389/tcp 443/tcp 80/tcp 445/tcp 137/any",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 445/tcp",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 443/tcp 80/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 4443/tcp"
                ],
                "Allow": [
                    "fake.domain.local",
                    "fake.domain.local"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "192.192.192.2/24"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 443/tcp",
                    "fake.domain.local 443/tcp 22/tcp",
                    "fake.domain.local 22/tcp 443/tcp 3306/tcp",
                    "fake.domain.local 22/tcp 443/tcp ",
                    "fake.domain.local 443/tcp 22/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp 443/tcp",
                    "fake.domain.local 3389/tcp 1433/tcp",
                    "fake.domain.local 3317/tcp 22/tcp",
                    "fake.domain.local 3306/tcp 22/tcp",
                    "fake.domain.local 3306/tcp 22/tcp 80/tcp 443/tcp",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local",
                    "fake.domain.local 10085/tcp",
                    "fake.domain.local 22/tcp 443/tcp",
                    "fake.domain.local 22/tcp 443/tcp",
                    "fake.domain.local 22/tcp 443/tcp 80/tcp",
                    "fake.domain.local 1433/tcp 3389/tcp",
                    "fake.domain.local 443/tcp",
                    "fake.domain.local",
                    "fake.domain.local 3389/tcp",
                    "192.192.192.2/22 3900/any",
                    "192.192.192.2/24 443/tcp 22/tcp 4444/tcp"
                ],
                "Allow": [
                    "192.192.192.2/24"
                ]
            },
            "group:ita": {
                "Mfa": [
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 22/tcp",
                    "fake.domain.local 22/tcp",
                    "fake.domain.local 22/tcp",
                    "fake.domain.local 22/tcp",
                    "fake.domain.local 4444/tcp",
                    "fake.domain.local 4444/tcp",
                    "fake.domain.local 4444/tcp",
                    "fake.domain.local 4444/tcp",
                    "fake.domain.local 4444/tcp 443/tcp",
                    "192.192.192.2/24",
                    "192.192.192.2/24",
                    "192.192.192.2/24"
                ]
            },
            "group:ita2": {
                "Mfa": [
                    "fake.domain.local 3389/tcp"
                ],
                "Allow": [
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 443/tcp",
                    "fake.domain.local",
                    "192.192.192.2/24",
                    "fake.domain.local 3389/any"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 445/tcp"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 3389/tcp"
                ]
            },
            "group:replaced": {
                "Allow": [
                    "fake.domain.local 3389/any",
                    "192.192.192.2 80/tcp",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.local 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 80/tcp 443/tcp",
                    "fake.domain.local 443/tcp"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local",
                    "fake.domain.local",
                    "fake.domain.localr"
                ]
            },
            "group:rdp-clients": {
                "Mfa": [
                    "fake.domain.local 3389/tcp",
                    "fake.domain.local 3389/tcp"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 443/tcp"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 443/tcp",
                    "fake.domain.local"
                ]
            },
            "group:replaced": {
                "Mfa": [
                    "fake.domain.local 3389/tcp"
                ]
            }
        }
    }

@NHAS
Copy link
Owner

NHAS commented Jan 12, 2024

Thanks for that! Really glad I could get significant performance improvement sorted it was quite fun.

It's awesome to see that you're using Wag successfully in a professional environment. I'd just like to let you know that I do accept donations on my support page and cryptocurrency wallets listed below, which helps keep this work going, and I tend to expedite issues opened by supporters.

Monero (XMR):
8A8TRqsBKpMMabvt5RxMhCFWcuCSZqGV5L849XQndZB4bcbgkenH8KWJUXinYbF6ySGBznLsunrd1WA8YNPiejGp3FFfPND

Bitcoin (BTC):
bc1qm9e9sfrm7l7tnq982nrm6khnsfdlay07h0dxfr

@NHAS
Copy link
Owner

NHAS commented Jan 21, 2024

A note on this, I have just found a security bug that occurs now as the map is not being recreated. Updating routes will fail to remove old routes from a users firewall. Thus leaving them able to hit routes they otherwise should not be able to.

As I am currently in the middle of a huge change from moving to sqlite3 to etcd for high availability I cannot go back and fix this right at the second.

@bluecraank
Copy link
Contributor Author

Thanks for the info. Do you have any commit which fix it?

@NHAS NHAS closed this as completed May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs testing (on unstable) If the feature exists, but isnt yet released
Projects
None yet
Development

No branches or pull requests

2 participants