Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Local DNS records are synced when they shouldn't be #80

Closed
MJVerhulst opened this issue Nov 10, 2022 · 12 comments · Fixed by #87
Closed

Local DNS records are synced when they shouldn't be #80

MJVerhulst opened this issue Nov 10, 2022 · 12 comments · Fixed by #87
Assignees
@mattwebbio

Comments

@MJVerhulst
Copy link

MJVerhulst commented Nov 10, 2022 •
edited

What happened?

I'm syncing two piholes (two docker containers), but do not want them to sync local DNS records, because of the different networks they operate in.

However, the SYNC_LOCALDNSRECORDS is being ignored. I've tried using
SYNC_LOCALDNSRECORDS: 'false' and SYNC_LOCALDNSRECORDS: 0, to no avail.

This is my config:

  orbital-sync-2:
    image: mattwebbio/orbital-sync:1
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1440

Version

latest

Log output

10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Signing in to http://192.168.1.200/admin...
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ✔️ Successfully signed in to http://192.168.1.200/admin!
10-11-2022 10:59:55
11/10/2022, 10:59:55 AM: ➡️ Downloading backup from http://192.168.1.200/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Backup from http://192.168.1.200/admin completed!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Signing in to http://192.168.1.202/admin...
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ✔️ Successfully signed in to http://192.168.1.202/admin!
10-11-2022 10:59:56
11/10/2022, 10:59:56 AM: ➡️ Uploading backup to http://192.168.1.202/admin...
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: ✔️ Success: 1/1 hosts synced.
10-11-2022 11:00:19
11/10/2022, 11:00:19 AM: Waiting 1400 minutes...
@cbundy
Copy link
Contributor

cbundy commented Nov 10, 2022

Has this changed for you in a recent version?

If you run with v.1.3.0 do you have the same issue?

@mattwebbio
Copy link
Owner

mattwebbio commented Nov 11, 2022 •
edited

In addition to @cbundy's comment, if you could add VERBOSE: true to your configuration and re-post the log output it would be super helpful :)

The server's response from the upload should get logged when verbose mode is enabled.

@MJVerhulst
Copy link
Author

I copied and pasted from the homepage so I was apparently running version 1. Nevertheless, there seems to be no difference between versions. See below.

Just to be sure, I've been running two instances/containers of this image, since I want to sync my primary pi-hole completely to a second pihole, and then without to DNS-records to another pihole. Maybe things (like the backup.zip) are getting mixed up on the host file-system? I suppose this shouldn't happen, but just to make sure.
Complete docker-compose:

  orbital-sync:
    image: mattwebbio/orbital-sync:1.4.0
    container_name: pihole-sync
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.20'
      SECONDARY_HOST_1_PASSWORD: ''
      UPDATE_GRAVITY: 'false'
      INTERVAL_MINUTES: 1440
      TZ: 'Europe/Amsterdam'
  orbital-sync-2:
    image: mattwebbio/orbital-sync:1.3.0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output for 1.3.0

11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Backup uploaded to http://192.168.1.202!
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: Result:
11-11-2022 7:51:32
Processed adlist (31 entries)<br>
11-11-2022 7:51:32
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:51:32
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:51:32
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:51:32
Processed client (3 entries)<br>
11-11-2022 7:51:32
Processed client group assignments (5 entries)<br>
11-11-2022 7:51:32
Processed local DNS records (16 entries)<br>
11-11-2022 7:51:32
Processed local CNAME records (23 entries)<br>
11-11-2022 7:51:32
Processed domain_audit (0 entries)<br>
11-11-2022 7:51:32
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:51:32
Processed group (2 entries)<br>
11-11-2022 7:51:32
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:51:32
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:51:32
OK
11-11-2022 7:51:32
11/11/2022, 7:51:32 AM: ✔️ Success: 1/1 hosts synced.

Output for 1.4.0.

11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Backup uploaded to http://192.168.1.202/admin!
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Result:
11-11-2022 7:45:34
Processed adlist (31 entries)<br>
11-11-2022 7:45:34
Processed adlist group assignments (31 entries)<br>
11-11-2022 7:45:34
Processed blacklist (exact) (8 entries)<br>
11-11-2022 7:45:34
Processed blacklist (regex) (0 entries)<br>
11-11-2022 7:45:34
Processed client (3 entries)<br>
11-11-2022 7:45:34
Processed client group assignments (5 entries)<br>
11-11-2022 7:45:34
Processed local DNS records (16 entries)<br>
11-11-2022 7:45:34
Processed local CNAME records (23 entries)<br>
11-11-2022 7:45:34
Processed domain_audit (0 entries)<br>
11-11-2022 7:45:34
Processed black-/whitelist group assignments (274 entries)<br>
11-11-2022 7:45:34
Processed group (2 entries)<br>
11-11-2022 7:45:34
Processed whitelist (exact) (261 entries)<br>
11-11-2022 7:45:34
Processed whitelist (regex) (3 entries)<br>
11-11-2022 7:45:34
OK
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: ✔️ Success: 1/1 hosts synced.
11-11-2022 7:45:34
11/11/2022, 7:45:34 AM: Waiting 1400 minutes...

@MJVerhulst
Copy link
Author

I've been experimenting a bit more, and it seems that also Environment variables like SYNC_LOCALCNAMERECORDS, SYNC_WHITELIST, are not getting picked up on. Which is interesting because UPDATE_GRAVITY: 'false', is getting handled accordingly.

No differences between versions 1.3.0 and 1.4.0.

@mattwebbio
Copy link
Owner

So sorry for the delay on this! I'm in the process of moving and so I've not had time to look into this further.

Orbital just tries to emulate the form inputs on the "restore" section of the teleporter - it's possible these have either changed or I just didn't set them up correctly in the first place.

My default troubleshooting step would be to manually run a backup and then try unchecking the things you don't want synced. If it works by hand, it's definitely an issue with Orbital's handling of the checkboxes and I can fix (or PRs welcome!)

@cbundy
Copy link
Contributor

cbundy commented Nov 20, 2022

I'm not sure if specifying false for those form properties is intended to work in pihole, but what I can see is that when I untick options for restore, the entire property is gone.

All ticked
image

Untick localdns and localcname
image

@MJVerhulst
Copy link
Author

Forgive me if this doesn't make any sense, I'm a bit guessing on how this should work. In my understanding, interpreting @cbundy response:

  1. Pihole does not want a variable=false when the form is submitted, it expects the value instead of being 'true', to be gone from posted form.
  2. Orbital-sync processes the variables in the generateForm() function:
  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

    form.append('whitelist', Config.syncOptions.whitelist);
    form.append('regex_whitelist', Config.syncOptions.regexWhitelist);
    form.append('blacklist', Config.syncOptions.blacklist);
    form.append('regexlist', Config.syncOptions.regexlist);
    form.append('adlist', Config.syncOptions.adlist);
    form.append('client', Config.syncOptions.client);
    form.append('group', Config.syncOptions.group);
    form.append('auditlog', Config.syncOptions.auditlog);
    form.append('staticdhcpleases', Config.syncOptions.staticdhcpleases);
    form.append('localdnsrecords', Config.syncOptions.localdnsrecords);
    form.append('localcnamerecords', Config.syncOptions.localcnamerecords);
    form.append('flushtables', Config.syncOptions.flushtables);

It may be the case that pihole does not even check the contents of the localdnsrecords variable, but simple checks whether the variable is present.

So, to solve the issue, the generateForm() function should be modified as to only add the variables that are not disabled through the environment-variables passed on from docker-compose.

In pseudo-code (I only know PHP a bit) something like this?

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);

 variables = array ('whitelist','regex_whitelist','blacklist','...');
 foreach (variables as $variable) {
    if(Config.syncOptions$variable!== 'false) {
          form.append('$variable', Config.syncOptions.$variable);
    }
 }

@cbundy
Copy link
Contributor

cbundy commented Nov 23, 2022 •
edited

Yep, I think you're spot on. I will give it a test this weekend and can put a PR together if you can't.

If you want to have a crack, there is a dev container so you can just start coding if you have docker installed, or you can code in the cloud as well without needing anything installed, just click this button above the code!

image

@MJVerhulst
Copy link
Author

MJVerhulst commented Nov 23, 2022 •
edited

I actually managed to use this cloud coding and created a pull request. Not sure if I should have done that, as I'm not able to use the variable name for the config.Syncoptions, it creates an "Identifier expected" error. In PHP using variables as variable names is relatively easy, in JS I've read that adding brackets should yield the same effect, but no luck so far.

  private generateForm(): typeof FormData.prototype {
    const form = new FormData();
    form.append('token', this.token);
    const variables = ['whitelist','regex_whitelist','blacklist','regexlist','adlist','client','group','auditlog','staticdhcpleases','localdnsrecords','localcnamerecords','flushtables'];
    let fLen = variables.length;

    for (let i = 0; i < fLen; i++) {
      var configVariable = variables[i];
      if(Config.syncOptions.[configVariable] != false) {
        form.append(variables[i],Config.syncOptions.[configVariable]);
      }
    }

    return form;
  }

See also 684386c

@mattwebbio mattwebbio mentioned this issue Nov 25, 2022
Merged
@mattwebbio
Copy link
Owner

mattwebbio commented Nov 25, 2022 •
edited

Thanks so much @cbundy and @MJVerhulst for looking in to this while I've been away! I really, really appreciate it. I saw you opened a PR @MJVerhulst but it didn't look like I had write access to the branch to update the tests. I've pushed up a similar PR that omits the form fields (rather than setting them to false), and I've pushed up a prerelease image:

mattwebbio/orbital-sync:1.4.1-0

@MJVerhulst if you have the opportunity to test, that would be amazing!

@MJVerhulst
Copy link
Author

MJVerhulst commented Nov 25, 2022 •
edited

Thanks @cbundy for your feedback on my JS "code" that you posted on the pull-request, very helpful :).

I did some testing and I think it's now working as expected :).

Thanks for the new version @mattwebbio!

Fiddled around a bit, output/config below. Apologies for the messed up/double time-stamps, dozzle does that appearently,

Check 1
Config

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'true'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'
      SYNC_WHITELIST: 'false'

Output

25-11-2022 19:32:06
11/25/2022, 7:32:06 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:32:07
11/25/2022, 7:32:07 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Result:
25-11-2022 19:32:30
Processed adlist (33 entries)<br>
25-11-2022 19:32:30
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:32:30
Processed blacklist (exact) (7 entries)<br>
25-11-2022 19:32:30
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:32:30
Processed client (26 entries)<br>
25-11-2022 19:32:30
Processed client group assignments (28 entries)<br>
25-11-2022 19:32:30
Processed local CNAME records (24 entries)<br>
25-11-2022 19:32:30
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:32:30
Processed group (2 entries)<br>
25-11-2022 19:32:30
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:32:30
OK
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:32:30
11/25/2022, 7:32:30 PM: Waiting 1400 minutes...

Check 2

  orbital-sync-2:       
    image: mattwebbio/orbital-sync:1.4.1-0
    container_name: pihole-sync-vpn
    environment:
      PRIMARY_HOST_BASE_URL: 'http://192.168.1.200'
      PRIMARY_HOST_PASSWORD: ''
      SECONDARY_HOST_1_BASE_URL: 'http://192.168.1.202'
      SECONDARY_HOST_1_PASSWORD: ''
      SYNC_LOCALDNSRECORDS: 'false' #different IPs
      SYNC_LOCALCNAMERECORDS: 'false'
      SYNC_WHITELIST: 'false'
      SYNC_REGEXLIST: 'true'
      SYNC_BLACKLIST: 'false'
      UPDATE_GRAVITY: 'false'
      TZ: 'Europe/Amsterdam'
      INTERVAL_MINUTES: 1400
      VERBOSE: 'true'

Output

25-11-2022 19:38:46
11/25/2022, 7:38:46 PM: ➡️ Signing in to http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.200/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Downloading backup from http://192.168.1.200/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Backup from http://192.168.1.200/admin completed!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Signing in to http://192.168.1.202/admin...
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ✔️ Successfully signed in to http://192.168.1.202/admin!
25-11-2022 19:38:47
11/25/2022, 7:38:47 PM: ➡️ Uploading backup to http://192.168.1.202/admin...
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Backup uploaded to http://192.168.1.202/admin!
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Result:
25-11-2022 19:38:56
Processed adlist (33 entries)<br>
25-11-2022 19:38:56
Processed adlist group assignments (33 entries)<br>
25-11-2022 19:38:56
Processed blacklist (regex) (0 entries)<br>
25-11-2022 19:38:56
Processed client (26 entries)<br>
25-11-2022 19:38:56
Processed client group assignments (28 entries)<br>
25-11-2022 19:38:56
Processed black-/whitelist group assignments (277 entries)<br>
25-11-2022 19:38:56
Processed group (2 entries)<br>
25-11-2022 19:38:56
Processed whitelist (regex) (3 entries)<br>
25-11-2022 19:38:56
OK
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: ✔️ Success: 1/1 hosts synced.
25-11-2022 19:38:56
11/25/2022, 7:38:56 PM: Waiting 1400 minutes...

@mattwebbio
Copy link
Owner

Wonderful! Thanks so much, @MJVerhulst and @cbundy :)

I've released this change with v1.5

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mattwebbio mattwebbio

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Omit false form fields when creating backup mattwebbio/orbital-sync
3 participants
@mattwebbio @MJVerhulst @cbundy