[feature] Added support for ZeroTier VPN backend #778

Aryamanz29 · 2023-06-20T08:35:37Z

Depends on:

[feature] Added support for ZeroTier to the OpenWRT backend #207 netjsonconfig#274

Additionally, basic support has been added for managing ZeroTier networks through OpenWISP using the ZeroTier Service API

Closes #604

coveralls · 2023-06-20T10:39:56Z

coverage: 98.992%. first build when pulling 3b9e0f7 on issue-604/add-zerotier-vpn-backend into 13a7965 on gsoc23.

pandafy

Good work @Aryamanz29! 👏🏼

This is still a draft, so it was expected to find shortcomings. I am documenting the issue I encountered below, so @Aryamanz29 can also address them before the PR is ready for review.

It is expected from user to put port along with hostname in the "Host" field of the VPN object (i.e. 127.0.0.1:9993). This is puzzling to me. I did a quick search but didn't find any setting to change the listen port for ZeroTier service API. Can you please verify this @Aryamanz29 ? If it is not possible to change the listen port on ZeroTier controller, then we can hard code this value.
If the configuration encounters any error, the page is reloaded, but the JS does not hide unnecessary fields.
Open for discussion: When creating a new ZeroTier VPN, there are certain fields which are read only. Thus, I don't see a point in rendering them.

These fields are automatically generated. I think, we should treat them as we treat key-pair values for OpenVPN and WireGuard. Set them as configuration variable at VPN level.
Use checkbox input for all boolean fields in JSONSchema (This needs to be done in NetJSONConfig)
I was not able to confirm this but something in ZeroTier VPN configuration is triggering "unsaved-changes.js" on the change page of existing VPN object.. When moving to a different webpage, the "unsaved changes" alert box shows up even when nothing is changed. @Aryamanz29 can you please confirm this?

openwisp_controller/config/base/vpn.py

openwisp_controller/config/static/config/js/vpn.js

openwisp_controller/config/api/zerotier_service.py

openwisp_controller/config/migrations/0048_alter_vpn_auth_token.py

Aryamanz29 · 2023-06-20T17:16:09Z

Meet Notes - 20th June

Explore the possibility of obtaining the Zerotier interface name.
Investigate how to retrieve Zerotier interface information using OpenWRT.
Determine if it's feasible to assign IP addresses to each Zerotier client via OpenWISP
Verify the configuration for multiple Zerotier networks in OpenWRT.
Explore the possibility of changing the Zerotier OpenWRT configuration to an interface format similar to WireGuard interface.
Consider converting read-only fields to system-defined variables.
Consider the subnet VPN model field when implementing IP assignment of Zerotier clients. Refer to this comment for more details.

openwisp_controller/config/api/zerotier_service.py

openwisp_controller/config/base/vpn.py

openwisp_controller/config/tasks.py

nemesifier

These are the notes taken during our last weekly meeting:

Use zerotier central controller as a model for naming UI fields, labels and help texts
When a zerotier VPN server is created we need to automate the addition of the server itself as a member to the zerotier network (let's use the internal IP address field)
Let's introduce a retry mechanism in case the API requests fail
Let's apply the same retry mechanism to all API calls, look at OpenWISP monitoring for inspiration
The retry mechanism should retry only the exceptions which are considered recoverable, that means not retrying when we get a 404, 403, 401 and any other status code which indicates that retrying will surely not fix the issue (but an admin has to intervene instead)
We need to introduce a new notification type to communicate the unrecoverable failures to the user, ideally we want to reuse this notification type for multiple background tasks, not just zerotier related

#778 (comment) - Handled the scenario where an invalid hostname is provided in the ZeroTier /status call. - Improved user experience by overriding the clean() method, which now reports that the auth_token field is required when the user selects the ZeroTier VPN backend. - Moved the logic from the _update_zerotier_server() method to the update_vpn_server_configuration method for better code organization. - Removed redundant conditional checks and unnecessary log statement from the _update_zerotier_server method.

- Added a new notification type for background API tasks. - This notification type will be used in the zerotier backend to notify users about various events related to background API calls. - Currently, the new notification is used in the following scenarios: - Creating a new zerotier network. - Updating an existing zerotier network. - Modifying network member configurations such as authorization and IP assignments.

**ZerotierService class** - Added a timeout of '5' seconds to API request methods. **OpenwispApiTask class** - Changed the log level to `logger.warn` when the retry limit is not reached, and `logger.error` only when the retry limit is reached. - Modified the logic to send a recovery notification only when the API task encountered an error before. Otherwise, no recovery notification is generated. - Updated the cache key as suggested in the review. - Added a docstring and comments for the `handle_api_call` method. **vpn.py** - Disabled email notifications for background API tasks. - Utilized guard clause in the `_validate_host` method. - Moved Zerotier VPN `auth_token` validation back to `models.py`. - Simplified the subnet and IP values for Zerotier API tasks. - Updated translation string messages and used the `.format` method instead of f-strings. For more information, see: https://docs.djangoproject.com/en/4.2/topics/i18n/translation/#standard-translation

- Added TestZeroTier and TestZeroTierTransaction - Updated `_invalidate_peer_cache` method of the Vpn model to emit `vpn_peers_changed` signal for only for Wireguard or VXLAN VPN backends

…calls - Ensure that recovery notifications are triggered only when the VPN server has previously encountered an unrecoverable error (error notification)

- Added a test case for zerotier vpn deletion 404

…pn server - (req-changes) Moved time import at the top of the file. - Subnet should be type cast after ip assignment evaluation. - Updated zt vpn deletion tests.

- Fixed test vpn deletion notification mock. - Deleted ZT API tasks notification cache keys after vpn deletion.

- Added exception handling to get_node_status() method. - Fixed timeout issues, added global constant for timeout. - Refactored zerotier network ip_start and ip_end assignments. - Added comment for update_vpn_server_configuration() method. - Updated signature of update_network_member() method for reusability with zt members. - Changed locmem.keys mock to settings.backned.keys - Moved subnet delete reset outside of subtest.

pandafy

LGTM!

…stency

Aryamanz29 added the enhancement label Jun 20, 2023

Aryamanz29 self-assigned this Jun 20, 2023

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch 3 times, most recently from 2c56913 to b2db3c7 Compare June 20, 2023 10:04

pandafy requested changes Jun 20, 2023

View reviewed changes

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch 5 times, most recently from 874c6ee to 1308e0d Compare June 29, 2023 11:33

Aryamanz29 requested a review from pandafy June 29, 2023 11:50

Aryamanz29 marked this pull request as ready for review June 29, 2023 11:50

pandafy requested changes Jun 30, 2023

View reviewed changes

nemesifier requested changes Jul 4, 2023

View reviewed changes

pandafy mentioned this pull request Jul 4, 2023

[change] Use retry mechanism for triggering VPN server webhook #785

Closed

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch from 9e8f310 to abc6a11 Compare July 8, 2023 16:39

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch 6 times, most recently from 4fa87d4 to 87b552d Compare July 17, 2023 17:04

Aryamanz29 added 16 commits August 11, 2023 16:37

[change] Updated zerotier auto_client method

19e9105

[fix] Fixed zerotier migration conflicts

758767f

[tests] Added zerotier vpn backend tests

27bc331

- Added TestZeroTier and TestZeroTierTransaction - Updated `_invalidate_peer_cache` method of the Vpn model to emit `vpn_peers_changed` signal for only for Wireguard or VXLAN VPN backends

[docs] Added how to setup zerotier tunnels

09131b6

[fix] Fixed zerotier server deletion 404

3dd2d7f

[req-change] Avoid sending recovery notifications after retrying API …

b7f3144

…calls - Ensure that recovery notifications are triggered only when the VPN server has previously encountered an unrecoverable error (error notification)

[tests] Improved zerotier vpn deletion tests

7e9a163

- Added a test case for zerotier vpn deletion 404

[tests] Added zerotier api tasks notifications tests

66a1afe

[changes] Controller node should leave network after deletion of zt v…

e93fe01

…pn server - (req-changes) Moved time import at the top of the file. - Subnet should be type cast after ip assignment evaluation. - Updated zt vpn deletion tests.

[deps] Updated netjsonconfig branch to gsoc23

da63f7b

- Fixed test vpn deletion notification mock. - Deleted ZT API tasks notification cache keys after vpn deletion.

[req-changes] Improved zerotier tutorial docs

2ddc642

[docs] Added OPENWISP_CONTROLLER_API_TASK_RETRY_OPTIONS

6f43c34

[fix] Fixed migration merge conflicts

c92b73e

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch 2 times, most recently from 69c3fd5 to 07b48f7 Compare August 11, 2023 11:54

nemesifier approved these changes Aug 17, 2023

View reviewed changes

pandafy approved these changes Aug 17, 2023

View reviewed changes

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch from 07b48f7 to ffd4eb1 Compare August 17, 2023 11:12

[req-changes] Moved zerotier tasks and changed 'exc' to 'e' for consi…

3b9e0f7

…stency

Aryamanz29 force-pushed the issue-604/add-zerotier-vpn-backend branch from ffd4eb1 to 3b9e0f7 Compare August 17, 2023 11:19

nemesifier merged commit 8dd834e into gsoc23 Aug 17, 2023
10 checks passed

nemesifier deleted the issue-604/add-zerotier-vpn-backend branch August 17, 2023 12:01

This was referenced Oct 11, 2023

[feature/change] Pre-generate ZeroTier Identity Keys in Advance Using OpenWISP #767

Closed

[feature] Allow managing ZeroTier networks from OpenWISP #605

Closed

[feature] Add a VPN backend to allow configuration of ZeroTier networks #765

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature] Added support for ZeroTier VPN backend #778

[feature] Added support for ZeroTier VPN backend #778

Aryamanz29 commented Jun 20, 2023 •

edited

Loading

coveralls commented Jun 20, 2023 •

edited

Loading

pandafy left a comment •

edited

Loading

Aryamanz29 commented Jun 20, 2023

nemesifier left a comment •

edited

Loading

pandafy left a comment

[feature] Added support for ZeroTier VPN backend #778

[feature] Added support for ZeroTier VPN backend #778

Conversation

Aryamanz29 commented Jun 20, 2023 • edited Loading

coveralls commented Jun 20, 2023 • edited Loading

pandafy left a comment • edited Loading

Choose a reason for hiding this comment

Aryamanz29 commented Jun 20, 2023

nemesifier left a comment • edited Loading

Choose a reason for hiding this comment

pandafy left a comment

Choose a reason for hiding this comment

Aryamanz29 commented Jun 20, 2023 •

edited

Loading

coveralls commented Jun 20, 2023 •

edited

Loading

pandafy left a comment •

edited

Loading

nemesifier left a comment •

edited

Loading