ec2-macos-init partially overwriting the userdata file

[agile-jtdressel]

I was experimenting with the capabilities of the Execute-User-Data module, and I did not want to wait for the dedicated host to be wiped between invocations.

• I created a mac ec2 instance with userdata in the aws web console.
• I edited the contents of /usr/local/aws/ec2-macos-init/instances/i-0a9977235e17ff7f1/userdata
• I changed the contents of /usr/local/aws/ec2-macos-init/init.toml to only run the Execute-User-Data module
• I ran sudo ec2-macos-init run

What I expected:

I expected ec2-macos-init to either run my newly edited userdata or to run the original userdata.

What happened:

ec2-macos-init seems to have merged the original userdata with my edited userdata, and then ran the merged userdata.

Versions

EC2 macOS Init
Version: 1.5.0 [2021-07-22 11:34:15 -0700]

macOS version: 11.5.1

Details

I started with the below userdata, entered in the aws web console.

#!/bin/bash
set -x #echo on

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

date -u +"%Y-%m-%dT%H:%M:%SZ" >> /tmp/launch.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/Shared/launch.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt

This worked pretty much how I expected. /tmp/launch.txt and /Users/Shared/launch.txt both existed and had reasonable timestamps in them. I don't recall, but I think /Users/ec2-user/Desktop/launch.txt did not exist after the first run. (I suspect it ran before the Desktop directory was created. )

I wanted to try a few edits of userdata to see how they were executed. (see #6 ). I had edited /usr/local/aws/ec2-macos-init/init.toml because some of the initial tasks failed. I believe those failed because I had changed the password for ec2-user.

# Default EC2 macOS Init init.toml config for mac1.metal instances

# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"). You
# may not use this file except in compliance with the License. A copy of
# the License is located at
#
# http://aws.amazon.com/apache2.0/
#
# or in the "license" file accompanying this file. This file is
# distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
# ANY KIND, either express or implied. See the License for the specific
# language governing permissions and limitations under the License.


### Group 4 ###
## Finally, run user data.

# Attempt to execute userdata, if provided
[[Module]]
    Name = "ExecuteUserData"
    PriorityGroup = 4 # Fourth group
    RunPerInstance = true # Run once per instance
    FatalOnError = false # Best effort, don't fatal on error
    [Module.UserData]
        ExecuteUserData = true # Execute the userdata

• I removed /usr/local/aws/ec2-macos-init/instances/i-0a9977235e17ff7f1/history.json.
• I made userdata writeable sudo chmod 755 userdata
• I edited the contents of userdata, then I made it read only and cated to verify my changes.

ec2-user@ip-172-31-26-0 i-0a9977235e17ff7f1 % sudo chmod 555 userdata
ec2-user@ip-172-31-26-0 i-0a9977235e17ff7f1 % ls -lah
total 8
drwxr-xr-x  3 root  wheel    96B Sep  2 22:30 .
drwxr-xr-x  3 root  wheel    96B Sep  2 18:58 ..
-r-xr-xr-x  1 root  wheel   338B Sep  2 22:27 userdata
ec2-user@ip-172-31-26-0 i-0a9977235e17ff7f1 % cat userdata
#!/bin/bash
set -x #echo on

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

date -u +"%Y-%m-%dT%H:%M:%SZ" >> /tmp/launch-4.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/Shared/launch-4.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch-4.txt

echo "this line is entirely new"
touch /tmp/last-line

ec2-user@ip-172-31-26-0 i-0a9977235e17ff7f1 % sudo ec2-macos-init run
2021/09/02 22:31:45.227416 Fetching instance ID from IMDS...
2021/09/02 22:31:45.229604 Running on instance i-0a9977235e17ff7f1
2021/09/02 22:31:45.229625 Reading init config...
2021/09/02 22:31:45.229994 Successfully read init config
2021/09/02 22:31:45.230006 Validating config...
2021/09/02 22:31:45.230031 Successfully validated config
2021/09/02 22:31:45.230060 Prioritizing modules...
2021/09/02 22:31:45.230064 Successfully prioritized modules
2021/09/02 22:31:45.230084 Creating instance history directories for current instance...
2021/09/02 22:31:45.230121 Successfully created directories
2021/09/02 22:31:45.230142 Getting instance history...
2021/09/02 22:31:45.230279 Successfully gathered instance history
2021/09/02 22:31:45.230292 Processing priority level 1 (0 modules)...
2021/09/02 22:31:45.230298 Successfully completed processing of priority level 1
2021/09/02 22:31:45.230302 Processing priority level 2 (0 modules)...
2021/09/02 22:31:45.230306 Successfully completed processing of priority level 2
2021/09/02 22:31:45.230309 Processing priority level 3 (0 modules)...
2021/09/02 22:31:45.230313 Successfully completed processing of priority level 3
2021/09/02 22:31:45.230337 Processing priority level 4 (1 modules)...
2021/09/02 22:31:45.230369 Running module [ExecuteUserData] (type: userdata, group: 4)
2021/09/02 22:31:45.242850 Successfully completed module [ExecuteUserData] (type: userdata, group: 4) with message: successfully ran user data with stdout: [this line is entirely new
] and stderr: [+++ dirname /usr/local/aws/ec2-macos-init/instances/i-0a9977235e17ff7f1/userdata
++ cd /usr/local/aws/ec2-macos-init/instances/i-0a9977235e17ff7f1
++ pwd
+ DIR=/usr/local/aws/ec2-macos-init/instances/i-0a9977235e17ff7f1
+ date -u +%Y-%m-%dT%H:%M:%SZ
+ date -u +%Y-%m-%dT%H:%M:%SZ
+ date -u +%Y-%m-%dT%H:%M:%SZ
+ echo 'this line is entirely new'
+ touch /tmp/last-line
]
2021/09/02 22:31:45.242902 Successfully completed processing of priority level 4
2021/09/02 22:31:45.242927 Writing instance history for instance i-0a9977235e17ff7f1...
2021/09/02 22:31:45.243164 Successfully wrote instance history
2021/09/02 22:31:45.243177 EC2 macOS Init completed in 13.551741ms

Afterward, I looked at the contents of userdata, and was surprised to see it was a mix of my original and edited userdata. Note the date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt-4.txt line, which is not in either the original or edited userdata.

#!/bin/bash
set -x #echo on

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"

date -u +"%Y-%m-%dT%H:%M:%SZ" >> /tmp/launch.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/Shared/launch.txt
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt-4.txt

echo "this line is entirely new"
touch /tmp/last-line

The combined contents were executed. /tmp/launch.txt, /Users/Shared/launch.txt, /Users/ec2-user/Desktop/launch.txt-4.txt and touch /tmp/last-line were all either created or updated.

[mattcataws]

Hey @agile-jtdressel, thanks for opening an issue to address these unexpected results you've been seeing.

My first guess as to why you might be seeing this behavior is that the userdata file on the instance got spliced when the UserDataModule ran. This probably happened due to the local edits to userdata while the original userdata tied to the instance remained unchanged. The UserDataModule then downloaded the original userdata and wrote it on top of the local changes that were made. But if this is the case then I'm curious as to why date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt-4.txt was not overwritten with date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt.

One thought I had is that we can prevent this by adding another option to the Execute-User-Data module to control the overwrite behavior. This option would toggle between overwriting with the latest and never overwriting. Please let me know your thoughts on this option and if it's something you think would be useful.

This is all just initial speculation based on the information you've provided. The issue is going to require a bit more investigation on our side in order to determine what's actually going on. We'll keep this issue updated as we learn more about the problem.

Also, please post any new data, comments, and/or questions you have along the way :).

[agile-jtdressel]

My first guess as to why you might be seeing this behavior is that the userdata file on the instance got spliced when the UserDataModule ran. This probably happened due to the local edits to userdata while the original userdata tied to the instance remained unchanged. The UserDataModule then downloaded the original userdata and wrote it on top of the local changes that were made. But if this is the case then I'm curious as to why date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt-4.txt was not overwritten with date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt.

I wish I saved the exact file (or dumped the hex values with xxd), I wonder if there's a weird end-of-file character issue here. Granted, I don't know how userdata is encoded, but I would have expected something like date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt.txt based on the number of characters

date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt # original userdata
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch-4.txt # manually updated userdata
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt-4.txt # actual result modifed by ec2-macos-init
date -u +"%Y-%m-%dT%H:%M:%SZ" >> /Users/ec2-user/Desktop/launch.txt.txt # what I would have expected based on the number of characters. 

One thought I had is that we can prevent this by adding another option to the Execute-User-Data module to control the overwrite behavior. This option would toggle between overwriting with the latest and never overwriting. Please let me know your thoughts on this option and if it's something you think would be useful.

This would be tremendously useful to me. I was also toying with the idea of adding an option with a path to an arbitrary userdata.

[mattcataws]

Thanks for your feedback about that idea! I'll make a note of the possible option while we look to resolve this issue.

I was also toying with the idea of adding an option with a path to an arbitrary userdata.

That's an interesting idea. What would your use case look like for this option?

[agile-jtdressel]

Thanks for your feedback about that idea! I'll make a note of the possible option while we look to resolve this issue.

I was also toying with the idea of adding an option with a path to an arbitrary userdata.

That's an interesting idea. What would your use case look like for this option?

I should note an assumption: that specifying the path would disable overwriting userdata.

It would be very similar to setting the Overwrite = never / Overwrite = always that you described. It would make it easier to experiment on a non-ec2 system. Being able to specify the path would make it easier to experiment on non-ec2 systems.

Right now, I think I could experiment with userdata in a local VM (or just on my development MacBook) by faking the metadata service and hosting a simple server on 169.254.169.254. If I could configure ec2-macos-init to read userdata from an arbitrary existing file, I could save some time spinning up a server and configuring it to serve my desired userdata.

Envisioned usage:

/usr/local/aws/ec2-macos-init/init.toml:

[[Module]]
  Name = "Execute-User-Data"
  PriorityGroup = 4 # Fourth group
  RunPerInstance = true # Run once per instance
  UserdataPath = /Users/jtdressel/ci_template/userdata_a
  FatalOnError = false # Best effort, don't fatal on error
  [Module.UserData]
    ExecuteUserData = true # Execute the userdata

[mattcataws]

Thanks for detailing the information about what that option might look like. Please do create a separate issue where we can discuss this option and your use case further.

[grahamc]

For what it is worth, for my purposes I just added another module at the end:

[[Module]]
    Name = "BogusUserData"
    PriorityGroup = 4 # Second group
    RunPerBoot = true
    FatalOnError = false # Best effort, don't fatal on error
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", "/usr/local/aws/ec2-macos-init/userdata"]

and iterated on that in place. Maybe this will be helpful for future explorers!