Fog (using foreman) integration to GCE

[DaveEzrakhovich]

Hey guys
I've been working with Foreman for some while and recently started working on GCE integration (We had EC2 integration for some while and its working properly)

I don't know if you're aware but foreman is using your library to work with various cloud providers

I've found several issues with the current library and GCE specifically

1. You don't support regions and subnetworks:
   Issue: Once a subnet is defined in foreman by default it assumes you have using the Google predefined "default" network and all the configuration comes with it.
   As of both VPN and security improvement.
   Solution: I had to change lib/fog/google/requests/compute/insert_server.rb file,
   Currently I've added hard coded section that looks at the following code:

     networkInterfaces = []
     if ! network.nil?
       networkInterface = {
         'network' => @api_url + @project + "/global/networks/#{network}",
         'subnetwork' => @api_url + @project + "/regions/<region_name>/subnetworks/<subnetwork_name>"
       }
       networkInterface['accessConfigs'] = [access_config] if access_config
       networkInterfaces << networkInterface
     end
   

The hard coded area specifies that we're using the region name (which is specified in the GCE plugin configuration) and subnetwork specified

1. custom METADATA:
   Issue: As google instances are blocked for root access, and therefor foreman is blocked of running finish script we had to open it at least temporarily, and the best way to do so (while maintaining the base google images, at least the RedHat and CentOS) was adding metadata for the GCE to run a script as the instance comes up.

Solution: I had to add the following def on lib/fog/google/models/compute/server.rb

    def allow_ssh
      if self.metadata.nil?
        self.metadata = Hash.new("")
      end

      if !self.metadata["startup-script"]
        self.metadata["startup-script"] = ""
      end

      if !self.metadata["startup-script"].empty?
        self.metadata["startup-script"] += "\n"
      end

      self.metadata["startup-script"] += "#!/bin/bash\n/bin/sed -i 's/PermitRootLogin no/PermitRootLogin yes/g' /etc/ssh/sshd_config\nservice sshd restart"
    end

and obviously call it later on in the code

I think we also need to add support for custom metadata for GCE
I know i'd use it with Foreman but i think it'll be used by others as well

Foreman info:
Site: https://www.theforeman.org/
GitHub repo: https://github.com/theforeman/theforeman.org
Version: 1.12.2 (Latest)

Fog info:
Version: fog-google-0.1.0 library (at least from what I've found in the lib directory)

[icco]

So it looks like you're using the old version of fog-google (https://github.com/theforeman/foreman/blob/27ae0a7da15b3b26e85645c4773d15ac3d4d5157/bundler.d/fog.rb). Could you try requiring fog-google (and fog-aws, you don't actually want to require normal fog unless you need 1.9 support), because I most of these issues have been fixed in more recent issues.

[icco]

cc @Temikus

[Temikus]

@DaveTaboola Did updating to recent fog-google help?

[icco]

No response. Feel free to reopen!

[shahaf600]

Hi, right now we are using foreman 1.14, and it's using fog-google-0.1.0 and google-api-client-0.8.2-5.
it does not support shared VPC of google, it's automatically using default network declared in the gcloud project, and it's even not considering if you are using a different subnet in foreman instead of the default gcloud project network.

[shahaf600]

created an issue about that, if you can relate please.
http://projects.theforeman.org/issues/23366

[icco]

@shahaf600 what you're describing are very old and unsupported versions of fog-google. We only support the most recent release.

[Temikus]

@shahaf600 For what it's worth, XPN support does exist in fog 2.0+ (one can specify a network project when manipulating networks), but it seems it's up to foreman to bump the dep and implement it.

If there are any specific questions on implementation, however - let me know and I'll try to help.

EDIT: Grammer

[shahaf600]

@Temikus ok got it, you are right it's foreman's responsibility, but we want to make a contact with them to support a more progressed release of fog, and I didn't get what you meant by "XPN support does exist in fog 2.0", because there are: fog, fog-core, fog-google. what does foreman need to use so they can manage projects in google cloud ? (what is the the first release that started to support XPN?)

[Temikus]

@shahaf600 Answering very quickly so forgive me for brevity/typos.

Sorry, should've been a bit more clear. XPN support exists in fog-google since a while ago, one can specify network_interfaces array with a network reference from any project while creating an instance, you can reference a recent PR to vagrant to see how it's done (they use fog-google as a driver): mitchellh/vagrant-google#182

I'm not sure what's the minimum version for that but in any case highly recommend using fog-google >1.2 since the older version of google-api-client (which was a dependency) is no longer maintained.

fog is the old metagem and should not be used unless you depend on a provider that has not yet been separated, see fog/fog#3942, so just requiring fog-google if you need google support is enough.

fog-core is the separated core modules, they will be picked up as a dependency by fog-google automatically.

Makes sense? If not - no worries - just let me know and I'll try to come up with a self-contained example :)