Problems creating geolocation databases in "airgapped" environment

[MatthewJohn]

Describe the problem

I am running netbird (or at least, getting there) in an airgapped environment (in the sense of lacking access to the internet).

I was trying to get the geolocation database working and saw https://docs.netbird.io/selfhosted/geo-support. Although it gave the download paths for the source data, it didn't say how to use it.

I tried several things (which felt obvious), placing the downloaded artifact into the netbird data dir, but this didn't help.
After reading the code, I found it a little confusing...

1. NewGeolocation is called.
2. A "glob" using mmdbGlobPattern and geonamesDbGlobPattern, along with the download URLs is passed to getDatabaseFilename:
   • If the glob for pre-existing database matches, it returns this file (assuming autoUpdate is disabled), otherwise it calls getFilenameFromURL to hit the upstream to obtian the download "source" file (e.g. GeoLite-City-CSV_YYYYMMDD.tar.gz), strips the tar.gz and returns some file that will (eventually) be created as the DB files.
3. Then NewGeolocations calls loadGeolocationDatabases, which if the resulting DB file exists, skips doing anything, ortherwise it downloads, extracts and then calls a callback to process the file.

Unfortunately, nearly all of the actual functionality is performed in loadGeolocationDatabases, which only changes in functionality (in terms of code-path) depending on whether the generated DB exists or not.
I was wondering about re-working this and creating a PR, so it ran in more of a:

1. Check if DB file exists or check if auto-updating
2. Check if source file exists, else download them (or force if auto-updating)
3. Extract/import

Unfortunately, for me, both getDatabaseFilename and loadGeolocationDatabases both connect to the upstream. I also noted that the upstream URLs were unconfigurable (another option I was toying with changing).

I was really strugglig to get netbird running locally (due to SSL certs and the amount of automation that assumes that DNS+public inbound connections are available) to be able to just copy the files generated from another instance..

However, I realised that I could just create a simple script to do this locally:

package main

import (
	"context"
	"log"
	"os"

	"github.com/netbirdio/netbird/management/server/geolocation"
)

func main() {
	tempDir, err := os.MkdirTemp("", "example")
	if err != nil {
		log.Fatal(err)
	}
	log.Print(tempDir)
	_, err = geolocation.NewGeolocation(context.Background(), tempDir, true)
	if err != nil {
		log.Fatal(err)
	}

}

Whilst it worked and I could upload the generated files to an artifact store.. I'd need to create a pipeline to capture this code, pull netbird source, build and push into the artifact store, which isn't great.

Perhaps if anyone else has this issue, the code snippit above might help, but otherwise, I'm wondering if the suggestions I made around handling pre-existing zip/tar files and/or configurations to override the zip/tar.gz file URLs would be something that could be of interest?

To Reproduce

Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior

Netbird can run without internet access and be able to generate geolocation databases.

Are you using NetBird Cloud?

Self-hosted

NetBird version

netbird version

Is any other VPN software installed?

N/a

Debug output

N/a

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Have you tried these troubleshooting steps?

• Checked for newer NetBird versions
• Searched for similar issues on GitHub (including closed ones)
• Restarted the NetBird client
• Disabled other VPN software
• Checked firewall settings

[iWangJiaxiang]

Hi, could you plz share the generated files here. Just drag & drop into comment. It would be helpful