Config.includes not picked up when client repacking

[OmarDarwish]

I’m trying to let clients add labels via files. This is part of the config for my frontends that are doing the client repacking:

Client Context:
  Platform:Darwin:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Linux:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Windows:
    Config.includes:
      - build.yaml
      - "%(Client.install_path)/%(Client.binary_name).labels.yaml"

But when installing the mac .pkg, the Config.includes is never picked up:

$ cat /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml                                                                                                                                                                                                                                                                      
Client.arch: amd64
Client.company_name: GRR Project
Client.description: '%(name) %(platform) %(arch)'
Client.foreman_check_frequency: 1800
Client.install_path: /usr/local/lib/%(Client.name)/%(ClientRepacker.output_basename)
Client.name: grr
Client.platform: darwin
Client.plist_filename: '%(Client.plist_label).plist'
Client.plist_label: '%(Client.plist_label_prefix).google.code.%(Client.name)'
Client.plist_label_prefix: com
Client.plist_path: /Library/LaunchDaemons/%(Client.plist_filename)
Client.poll_max: 600
Client.rekall_profile_cache_path: '%(Client.install_path)/rekall_profiles'
Config.includes:
- build.yaml
Config.writeback: /etc/%(Client.name).local.yaml
Logging.engines: stderr,file,syslog
Logging.path: /var/log
Logging.syslog_path: /var/run/syslog
Logging.verbose: false
Client.deploy_time: '2018-11-01 07:57:29'

The repacking is using the right contexts and is picking up Config.includes, but I have no idea where /tmp/tmpZFFwpY/grr.yaml is coming from. Maybe it comes from the pre-baked OSX template?

Repacking template: /usr/share/grr-server/grr-response-templates/templates/grr_3.2.4.3_amd64.xar.zip
DEBUG:2018-12-28 22:08:18,114 8 MainProcess 140222624716544 MainThread config_lib:682] Applying filter env for CLIENT_INSTALLER_FINGERPRINT.
Using context: [u'ClientBuilder Context', u'ClientBuilder Context', u'Arch:amd64', u'Platform:Darwin', u'Target:Darwin', u'Target:Darwin'] and labels: []
DEBUG:2018-12-28 22:08:18,116 8 MainProcess 140222624716544 MainThread config_lib:1160] Loading configuration from /tmp/tmpZFFwpY/grr.yaml
DEBUG:2018-12-28 22:08:18,117 8 MainProcess 140222624716544 MainThread config_lib:850] Configuration writeback is set to /tmp/tmpZFFwpY/grr.yaml
...
DEBUG:2018-12-28 22:08:18,126 8 MainProcess 140222624716544 MainThread build:321] Copying config option to client: Config.includes
...
INFO:2018-12-28 22:08:18,131 8 MainProcess 140222624716544 MainThread config_lib:501] Writing back configuration to file /tmp/tmpZFFwpY/grr.yaml

[grrrrrrrrr]

Hey Omar,

I played with this a bit and everything works as expected for me. If I add the context for Platform:Darwin to my grr-server.yaml, I get the expected Config.includes in the resulting client installer.

The tempfile is a red herring. We create the config that goes into the template by writing to a temp directory and reading it back.

Before I look more into this, can you make sure that other changes are properly propagated to the client? You could for example change the Client.company_name and see if that changes anything?

One other thing I noticed, you mentioned this issue end of December (sorry for the wait to pick up on it :)) but your client shows a deploy time (==time of repacking) of November 1st. Could it be that you are looking at some old client installer?

[OmarDarwish]

I just reran repacking inside of a container:

  $GRR_VENV/bin/grr_config_updater \
    --secondary_configs /etc/grr/server.local.yaml \
    -p Logging.verbose=True \
    -p Client.company_name="omar-test-company" \
    repack_clients

This is the a snippet from /etc/grr/server.local.yaml:

Client.name: omar
Client.company_name: omars-company

ClientBuilder Context:
  Platform:Darwin:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Linux:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Windows:
    Config.includes:
      - build.yaml
      - "%(Client.install_path)/%(Client.binary_name).labels.yaml"

And then tried to install the resultant pkg:

# After running repacking in the container
bash-3.2$ docker exec grr-frontend ls -l /usr/share/grr-server/executables/installers
total 178628
-rw-r--r-- 1 root root 21446505 Jan  8 21:32 dbg_omar_3.2.4.3_amd64_docker.exe
-rw-r--r-- 1 root root 19590186 Jan  8 21:32 dbg_omar_3.2.4.3_i386_docker.exe
-rw-r--r-- 1 root root      638 Jan  8 21:32 grr_3.2.4.3_amd64_docker.changes
-rw-r--r-- 1 root root 20450304 Jan  8 21:32 grr_3.2.4.3_amd64_docker.deb
-rw-r--r-- 1 root root 20972672 Jan  8 21:33 grr_3.2.4.3_amd64_docker.pkg
-rw-r--r-- 1 root root 21323951 Jan  8 21:33 grr_3.2.4.3_amd64_docker.rpm
-rw-r--r-- 1 root root      634 Jan  8 21:33 grr_3.2.4.3_i386_docker.changes
-rw-r--r-- 1 root root 18027860 Jan  8 21:33 grr_3.2.4.3_i386_docker.deb
-rw-r--r-- 1 root root 20044062 Jan  8 21:32 grr_3.2.4.3_i386_docker.rpm
-rw-r--r-- 1 root root 21446428 Jan  8 21:32 omar_3.2.4.3_amd64_docker.exe
-rw-r--r-- 1 root root 19590112 Jan  8 21:32 omar_3.2.4.3_i386_docker.exe

# Removing any remnents of grr from local machine
bash-3.2$ sudo rm -rf /usr/local/lib/grr
bash-3.2$ ls /usr/local/lib/grr
ls: /usr/local/lib/grr: No such file or directory

# Copying the installer from the container to local machine
bash-3.2$ docker cp grr-frontend:/usr/share/grr-server/executables/installers/grr_3.2.4.3_amd64_docker.pkg /tmp/installer.pkg
bash-3.2$ ls -l /tmp/installer.pkg
-rw-r--r--  1 omard  wheel  20972672 Jan  8 16:33 /tmp/installer.pkg

# Runn the installer
bash-3.2$ sudo installer -pkg /tmp/installer.pkg -target /
installer: Package name is grr
installer: Upgrading at base path /
installer: The upgrade was successful.

# Looking at the just installed grr config
bash-3.2$ cat /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml
Client.arch: amd64
Client.company_name: GRR Project
Client.description: '%(name) %(platform) %(arch)'
Client.foreman_check_frequency: 1800
Client.install_path: /usr/local/lib/%(Client.name)/%(ClientRepacker.output_basename)
Client.name: grr
Client.platform: darwin
Client.plist_filename: '%(Client.plist_label).plist'
Client.plist_label: '%(Client.plist_label_prefix).google.code.%(Client.name)'
Client.plist_label_prefix: com
Client.plist_path: /Library/LaunchDaemons/%(Client.plist_filename)
Client.poll_max: 600
Client.rekall_profile_cache_path: '%(Client.install_path)/rekall_profiles'
Config.includes:
- build.yaml
Config.writeback: /etc/%(Client.name).local.yaml
Logging.engines: stderr,file,syslog
Logging.path: /var/log
Logging.syslog_path: /var/run/syslog
Logging.verbose: false
Client.deploy_time: '2018-11-01 07:57:29'

[OmarDarwish]

Seems like the configuration file is not being changed by the client packing process?

bash-3.2$ ls -l /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml
-rwxr-xr-x  1 root  wheel  830 Nov  1 03:57 /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml

[grrrrrrrrr]

I think we went away from versioned directories. The correct path should be something like /usr/local/lib/grr/grr.yaml

[OmarDarwish]

bash-3.2$ ls /usr/local/lib/grr
grr_3.2.4.3_amd64

[OmarDarwish]

I played with this a bit and everything works as expected for me. If I add the context for Platform:Darwin to my grr-server.yaml, I get the expected Config.includes in the resulting client installer.

I might be doing something wrong.

• What does your grr-server.yaml look like?
• What command did you run for repacking?
• What version are you running?
• What environment are you running repacking in?
• How did you check the resulting configuration?

[grrrrrrrrr]

Can you try to unzip your grr_3.2.4.3_amd64_docker.pkg? Should have build.yaml and config.yaml.

[OmarDarwish]

bash-3.2$ unzip installer.pkg
Archive:  installer.pkg
 extracting: build.yaml
 extracting: config.yaml

bash-3.2$ cat config.yaml

...
Config.includes:
- build.yaml
...

[grrrrrrrrr]

And deploy time is current? Can you try changing one of the other options that are definitely copied to that config and repack again?

[OmarDarwish]

Deploy time is not current. I tried changing client name and company but no change is picked up. Does config updater not use the -p or --secondary_configs flags?

Should I be repacking with a different command?

[grrrrrrrrr]

hm maybe it doesn't use the secondary config? I put the changes into grr-server.yaml directly.

However, regardless of what options the config updater uses, the deploy time will always be the current timestamp. I can't really see a case where the file timestamp is current but the deploy timestamp is not?

What you can also do is use grr_client_build, for example like

grr_client_build repack --template env/grr-response-templates/templates/grr_3.2.4.6_amd64.rpm.zip --output_dir /tmp/out

[OmarDarwish]

Deploy time is not current.

I overlooked the full config.yaml after unzipping. It does in fact have the current Client.deploy_time, so I tried something new.

I modified /etc/grr/server.local.yaml with the following:

Client.name: omar
Client.company_name: omars-company

Platform:Darwin:
  Client.name: omar-darwin-test
  Config.includes:
    - "/etc/%(Client.name).labels.yaml"
    - build.yaml

And then ran

  $GRR_VENV/bin/grr_config_updater \
    --secondary_configs /etc/grr/server.local.yaml \
    -p Logging.verbose=True \
    -p Client.executable_signing_public_key="$CLIENT_EXECUTABLE_SIGNING_PUBLIC_KEY" \
    -p CA.certificate="$CA_CERTIFICATE" \
    -p Client.server_urls="http://$CLIENT_PACKING_FRONTEND_HOST:$FRONTEND_SERVER_PORT/" \
    repack_clients

Which produces

dbg_omar_3.2.4.3_amd64_docker.exe
dbg_omar_3.2.4.3_i386_docker.exe
grr_3.2.4.3_amd64_docker.changes
grr_3.2.4.3_amd64_docker.deb
grr_3.2.4.3_amd64_docker.rpm
grr_3.2.4.3_i386_docker.changes
grr_3.2.4.3_i386_docker.deb
grr_3.2.4.3_i386_docker.rpm
omar-darwin-test_3.2.4.3_amd64_docker.pkg
omar_3.2.4.3_amd64_docker.exe
omar_3.2.4.3_i386_docker.exe

So clearly grr_config_updater is aware of the secondary config. I then unzipped omar-darwin-test_3.2.4.3_amd64_docker.pkg and then cat config.yaml:

CA.certificate: '<CA cert>`
Client.company_name: omars-company
...
Client.executable_signing_public_key: ' <public key> `
...
Client.name: omar-darwin-test
Client.platform: darwin
...
Client.server_urls: http://frontend:8080/
...
Config.includes:
- build.yaml
...
Client.deploy_time: '2019-01-08 23:07:29'

So you can see the zipped config.yaml, contains everything but the additional Config.includes.

So I then installed the pkg. Nothing from the zipped config.yaml seems to get installed:

bash-3.2$ cat /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml
Client.arch: amd64
Client.company_name: GRR Project
Client.description: '%(name) %(platform) %(arch)'
Client.foreman_check_frequency: 1800
Client.install_path: /usr/local/lib/%(Client.name)/%(ClientRepacker.output_basename)
Client.name: grr
Client.platform: darwin
Client.plist_filename: '%(Client.plist_label).plist'
Client.plist_label: '%(Client.plist_label_prefix).google.code.%(Client.name)'
Client.plist_label_prefix: com
Client.plist_path: /Library/LaunchDaemons/%(Client.plist_filename)
Client.poll_max: 600
Client.rekall_profile_cache_path: '%(Client.install_path)/rekall_profiles'
Config.includes:
- build.yaml
Config.writeback: /etc/%(Client.name).local.yaml
Logging.engines: stderr,file,syslog
Logging.path: /var/log
Logging.syslog_path: /var/run/syslog
Logging.verbose: false
Client.deploy_time: '2018-11-01 07:57:29'

[grrrrrrrrr]

Since your client name has changed, you need to adjust the path to the yaml too. I'll try to do the same steps and see what happens.

[OmarDarwish]

you need to adjust the path to the yaml too

Which path?

[grrrrrrrrr]

Wherever it says grr in "/usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml" it will now be omar-darwin-test or omar.

[OmarDarwish]

This is from my mac after installing the latest .pkg.

Checking the .plist

bash-3.2$ cat /Library/LaunchDaemons/com.google.code.grr.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        ...
        <key>ProgramArguments</key>
        <array>
                <string>/usr/local/lib/grr/grr_3.2.4.3_amd64/grr</string>
                <string>--config=/usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml</string>
        </array>
        ...
</dict>
</plist>

Checking for any other config files

bash-3.2$ ls -l1 /usr/local/lib/grr/
grr_3.2.4.3_amd64

bash-3.2$ ls -l1 /usr/local/lib/grr/grr_3.2.4.3_amd64/ | grep .yaml
_yaml.so
build.yaml
grr.yaml
installer_config.yaml
libyaml-0.2.dylib

Contents of config file

bash-3.2$ cat /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml
Client.arch: amd64
Client.company_name: GRR Project
Client.description: '%(name) %(platform) %(arch)'
Client.foreman_check_frequency: 1800
Client.install_path: /usr/local/lib/%(Client.name)/%(ClientRepacker.output_basename)
Client.name: grr
Client.platform: darwin
Client.plist_filename: '%(Client.plist_label).plist'
Client.plist_label: '%(Client.plist_label_prefix).google.code.%(Client.name)'
Client.plist_label_prefix: com
Client.plist_path: /Library/LaunchDaemons/%(Client.plist_filename)
Client.poll_max: 600
Client.rekall_profile_cache_path: '%(Client.install_path)/rekall_profiles'
Config.includes:
- build.yaml
Config.writeback: /etc/%(Client.name).local.yaml
Logging.engines: stderr,file,syslog
Logging.path: /var/log
Logging.syslog_path: /var/run/syslog
Logging.verbose: false
Client.deploy_time: '2018-11-01 07:57:29'

[grrrrrrrrr]

I managed to reproduce with the commands you sent now. It works for me with a secondary config containing:

Target:Darwin:
Client.name: omar-darwin-test
Config.includes:
- "/etc/%(Client.name).labels.yaml"
- build.yaml

but wouldn't using Platform:Darwin: as the context. Does that work for you?

I think there is a conflict between the options, grr-server.yaml already has

Target:Darwin:
Config.includes:
- build.yaml

and then the config system can't determine which one to use since they have same depth. I'll fix this tomorrow by making build.yaml the default and removing that entry from the grr-server.yaml.

[OmarDarwish]

I tried rebuilding with Target:Darwin and now the config.yaml in the .pkg is picking up Config.includes properly, so progress!!

But when installling the .pkg, it's still using grr.yaml as the configuration, which doesn't contain any of my settings (see above).

Are you able to see your changes after installing?

[grrrrrrrrr]

Ok I understand now what's going on, this is all quite complicated :/

The grr.yaml is not supposed to change, it just copies this template from the installer and uses it as a base config. All your custom options come in /usr/local/lib/grr/grr_3.2.4.6_amd64/installer_config.yaml and then get written to the writeback file immediately - /etc/grr.local.yaml.

This, however probably doesn't work in your case either because the installer is a bit stricter than the client when opening configs and raises since it doesn't find the file you point it at. In case the installer fails, it will write a file /var/log/grr_installer.txt, you might see the corresponding backtrace in there.

To solve this, we could of course make the installer a bit more lenient when parsing the config but that might cause bugs later on.

Is all you want to set custom Client.labels based on a file? What you can do then is:

$ cat /etc/labels.txt
asd,qwe

config:
Client.labels: "%(/etc/labels|file)"

IN [2]: config.CONFIG["Client.labels"]
OUT[2]: [u"asd", u"qwe"]

[OmarDarwish]

I would like each client machine that is using this GRR client installer to be able to specify its own labels via a file at install/deploy time. That way each machine can segregate itself, and I don't have to anticipate, and repack clients with custom labels for every deploy.

I tried setting:

Target:Darwin:
  Client.labels: "%(/etc/grr-labels.txt|file)",

But this is being interpolated at repacking time, requiring the frontend to have this file exist, and is being evaluated as "Client.labels: Null" for the final client configuration at /etc/grr.local.yaml on my mac.

Can I tell it not to interpolate this value during repacking and just save it as is?

[OmarDarwish]

I then tried:

Client Context:
  Target:Darwin:
    Client.labels: "%(/etc/grr-labels.txt|file)",

and

Client Context:
  Platform:Darwin:
    Client.labels: "%(/etc/grr-labels.txt|file)",

and then having a /etc/grr-labels.txt with the contents hello,world.

After both repacking attempts, I unzipped the .pkg, and check the config.yaml. Both had the correct value of Client.labels: "%(/etc/grr-labels.txt|file)"

I then installed the .pkg, and checked the /etc/grr.local.yaml. It had Client.labels: null :(

[OmarDarwish]

I tried with Config.includes again.

All your custom options come in /usr/local/lib/grr/grr_3.2.4.6_amd64/installer_config.yaml

This doesn't appear to be the case with version 3.2.4.3 of grr.

and then get written to the writeback file immediately - /etc/grr.local.yaml.

This, however probably doesn't work in your case either because the installer is a bit stricter than the client when opening configs and raises since it doesn't find the file you point it at. In case the installer fails, it will write a file /var/log/grr_installer.txt, you might see the corresponding backtrace in there.

I checked this and you're right. When I run the installer without having a file /etc/grr.labels.yaml already present, the installer silently fails and leaves a stacktrace in /var/log/grr_installer.txt:

Starting installation procedure for GRR client.
Installation failed: Unable to load include file /etc/grr.labels.yaml
Traceback (most recent call last):
  File "site-packages/grr_response_client/installer.py", line 70, in RunInstaller
  File "site-packages/grr_response_core/lib/registry.py", line 271, in Init
  File "site-packages/grr_response_core/lib/registry.py", line 260, in _RunAllHooks
  File "site-packages/grr_response_core/lib/registry.py", line 243, in _RunSingleHook
  File "site-packages/grr_response_client/osx/installers.py", line 73, in Run
  File "site-packages/grr_response_client/osx/installers.py", line 40, in ExtractConfig
  File "site-packages/grr_response_core/lib/config_lib.py", line 1235, in Initialize
  File "site-packages/grr_response_core/lib/config_lib.py", line 1186, in LoadSecondaryConfig
ConfigFileNotFound: Unable to load include file /etc/grr.labels.yaml

So before I run installation, I create /etc/grr.labels.yaml with the following content:

Client.labels:
  - omar-label

I then reran the installer, and then cat /var/log/grr_installer.txt which only contains :

Starting installation procedure for GRR client.

So far so good. Lets see what /etc/grr.local.yaml looks like:

Config.includes:
- /etc/%(Client.name).labels.yaml
Client.labels: null
...

Yay! The includes made it in, but for some reason Client.labels is evaluating to null. I then manually ran sudo /usr/local/lib/grr/grr_3.2.4.3_amd64/grr --config /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml to see if it would change the configuration. It doesn't.

I then manually edited /etc/grr.local.yaml from Client.labels: null to Client.labels: omar-manuel and then reran grr manually as above. One of the log lines says:

INFO:2019-01-09 21:22:24,117 config_lib:501] Writing back configuration to file /etc/grr.local.yaml

So I double check /etc/grr.local.yaml again, and Client.labels was again set back to null!

I think at this point it's become a configuration context problem again. Any ideas? We're so close!

[grrrrrrrrr]

Ok I found an somewhat unrelated issue that would overwrite your labels. We have a few options that we always want to persist to the grr.local.yaml:

grr/grr/client/grr_response_client/client_startup.py

Line 34 in 36b8a2e

config.CONFIG.Persist("Client.labels")

The bug is that we assume that those options are always set in the main config (or not at all). In your case, it would take the unset main config and overwrite the custom config with it - hence the null in Yaml which is a Python None.

76dfefd fixes this bug, if you could try with a template after that commit, it should work now.

I also added an optionalfile config filter that will not raise if it can't find the file given. This is slightly dangerous since it will hide some errors but it should make this labeling process a bit easier.

[OmarDarwish]

Thanks, I'll try and install from source in a container. I would really appreciate a docker release including this if it's not too much hassle! 🙏

[OmarDarwish]

It looks like the travis osx build has been failing. Does this matter?

[grrrrrrrrr]

Hm the error looks unrelated, some homebrew issue. We'll have a look what's going on there.

[OmarDarwish]

I just pulled down the latest container and tried to do a client repack with grr_config_updater. It seems like requirements weren't updated?

Traceback (most recent call last):
  File "/usr/share/grr-server/bin/grr_config_updater", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3126, in <module>
    @_call_aside
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3110, in _call_aside
    f(*args, **kwargs)
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3139, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 583, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 596, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/share/grr-server/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 789, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (protobuf 3.3.0 (/usr/share/grr-server/lib/python2.7/site-packages), Requirement.parse('protobuf>=3.6.0'), set(['googleapis-common-protos']))

[OmarDarwish]

Workaround provided here

[OmarDarwish]

Your fixes worked! I can see my labels in /etc/grr.local.yaml after installing a client repacked with the code you pushed.

Thanks for looking into this!