Chef Client temporary file behaviour has changed in 13.7.16 (remote_file)

[ghost]

Description

Chef Client 13.7.16 does not clean up temporary files created by the remote_file resource until after the run is complete. This causes issues in some cases where the temporary files still exist when services are restarted. Previously the temporary files would be gone by the time the remote_file resource was done.

Chef Version

13.7.16

Platform Version

OL 7 / RHEL 7

Replication Case

You can use this recipe to recreate the issue. The initial run where it downloads the remote_file for the first time won't reproduce it, but subsequent runs, even if the file does not change, will reproduce it.

directory '/tmp/temp_file_test' do
  action :create
end

remote_file '/tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm' do
  action :create
  source 'https://packages.chef.io/files/stable/chef/13.7.16/el/7/chef-13.7.16-1.el7.x86_64.rpm'
end

execute 'list directory' do
  action :nothing
  command 'ls -la /tmp/temp_file_test/ > /tmp/chef_debug.txt 2>&1'
end

log 'temporary file for remote_file still exists' do
  action :write
  only_if 'ls -l /tmp/temp_file_test/.chef*'
  notifies :run, 'execute[list directory]', :immediately
end

Client Output

Starting Chef Client, version 13.7.16
resolving cookbooks for run list: ["temporary_file_test"]
Synchronizing Cookbooks:
  - temporary_file_test (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 4 resources
Recipe: temporary_file_test::default
  * directory[/tmp/temp_file_test] action create (up to date)
  * remote_file[/tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm] action create (up to date)
  * execute[list directory] action nothing (skipped due to action :nothing)
  * log[temporary file for remote_file still exists] action write

  * execute[list directory] action run
    - execute ls -la /tmp/temp_file_test/ > /tmp/chef_debug.txt 2>&1

Running handlers:
Running handlers complete
Chef Client finished, 2/5 resources updated in 02 seconds

cat /tmp/chef_debug.txt
total 47944
drwxrwxr-x.  2 vagrant vagrant     4096 Feb  1 18:11 .
drwxrwxrwt. 10 root    root        4096 Feb  1 18:11 ..
-rw-r--r--.  1 vagrant vagrant 49083487 Feb  1 18:10 chef-13.7.16-1.el7.x86_64.rpm
-rw-------.  1 vagrant vagrant        0 Feb  1 18:11 .chef-chef-1320180201-6159-dx2827.7.16-1.el7.x86_64.rpm

Chef 13.6.4 does not have this issue, it started in 13.7.16. Client Output for the previous version is included below. The log and execute resources don't trigger as there are no temporary files there after remote_file is done.

Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["temporary_file_test"]
Synchronizing Cookbooks:
  - temporary_file_test (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 4 resources
Recipe: temporary_file_test::default
  * directory[/tmp/temp_file_test] action create (up to date)
  * remote_file[/tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm] action create (up to date)
  * execute[list directory] action nothing (skipped due to action :nothing)
  * log[temporary file for remote_file still exists] action write (skipped due to only_if)

Running handlers:
Running handlers complete
Chef Client finished, 0/4 resources updated in 02 seconds

[lamont-granquist]

that'd be a side effect of switching the tempfile over to the one the rest of the file resources use. this should really get fixed consistently for all of them since there's no reason for this.

[lamont-granquist]

so #6701 was what changed. i'm real puzzled about the tempfile handling though since it should be explicitly getting cleaned up here: https://github.com/chef/chef/blob/master/lib/chef/provider/file.rb#L415-L417

[lamont-granquist]

oh the remote_file resource is up-to-date. that's curious.

[ghost]

Yeah, this happens even if the resource is not changing. It seems partially cleaned up, the file that hangs around until the end is always 0 bytes

[lamont-granquist]

i kind of suspect it only happens when the resource is not changing.

[ghost]

Yeah, it looks like that's the case...

[vagrant@layer201 ~]$ chef-client -z -r temporary_file_test
[2018-02-02T18:53:30+00:00] WARN: No config file found or specified on command line, using command line options.
Starting Chef Client, version 13.7.16
resolving cookbooks for run list: ["temporary_file_test"]
Synchronizing Cookbooks:
  - temporary_file_test (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 4 resources
Recipe: temporary_file_test::default
  * directory[/tmp/temp_file_test] action create (up to date)
  * remote_file[/tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm] action create
    - update content in file /tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm from 188266 to 89c96a
    (file sizes exceed 10000000 bytes, diff output suppressed)
    - restore selinux security context
  * execute[list directory] action nothing (skipped due to action :nothing)
  * log[temporary file for remote_file still exists] action write (skipped due to only_if)

Running handlers:
Running handlers complete
Chef Client finished, 1/4 resources updated in 17 seconds
[2018-02-02T18:53:48+00:00] WARN: No config file found or specified on command line, using command line options.


[vagrant@layer201 ~]$ chef-client -z -r temporary_file_test
[2018-02-02T18:53:54+00:00] WARN: No config file found or specified on command line, using command line options.
Starting Chef Client, version 13.7.16
resolving cookbooks for run list: ["temporary_file_test"]
Synchronizing Cookbooks:
  - temporary_file_test (0.1.0)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 4 resources
Recipe: temporary_file_test::default
  * directory[/tmp/temp_file_test] action create (up to date)
  * remote_file[/tmp/temp_file_test/chef-13.7.16-1.el7.x86_64.rpm] action create (up to date)
  * execute[list directory] action nothing (skipped due to action :nothing)
  * log[temporary file for remote_file still exists] action write

  * execute[list directory] action run
    - execute ls -la /tmp/temp_file_test/ > /tmp/chef_debug.txt 2>&1

Running handlers:
Running handlers complete
Chef Client finished, 2/5 resources updated in 10 seconds
[2018-02-02T18:54:05+00:00] WARN: No config file found or specified on command line, using command line options.

[lamont-granquist]

yeah so somehow when the cache control headers result in no file getting downloaded an empty file is created and then the returned tempfile pointer is nil. the way that the base file provider works is that the nil tempfile object means that it bypasses the management of the file contents (zero length files are valid to download, so the remote_file contents object can't hand back a tempfile to a zero length file). i don't understand how that is happening.

[lock]

This thread has been automatically locked because it has not had recent activity. Please open a new issue for related bugs and link to relevant comments in this thread.