Pasting the conversation from the Rails ticket over here since we've ascertained that this is a thor bug.

def inside(dir='', config={}, &block)
  verbose = config.fetch(:verbose, false)

  say_status :inside, dir, verbose
  shell.padding += 1 if verbose
  @destination_stack.push File.expand_path(dir, destination_root)

  FileUtils.mkdir_p(destination_root) unless File.exist?(destination_root) # <--- what about pretend??
  FileUtils.cd(destination_root) { block.arity == 1 ? yield(destination_root) : yield }

  @destination_stack.pop
  shell.padding -= 1 if verbose
end

--- from rails ticket ---
Ok good news is that it's definitely a Thor bug. Confirmed this with Jose. However he is too busy to work on this. Can you try and write the patch for Thor? The main thing to keep in mind is that in the pretend mode, the messages like —
create config/locales
create config/locales/en.yml
create config/boot.rb
create config/database.yml
Should still appear. So Jose has suggested that in pretend mode, you create the directory, pretend to create the files and then remove the directory.

-- end from rails ticket ---

I suppose the idea is to create the directory if it doesnt already exist and then remove it if it was indeed created by the thor action in the "inside" method.