Add scaleway provider #7331

Merged
merged 8 commits into from Jul 13, 2016

Projects

None yet

3 participants

@nicolai86
Collaborator
nicolai86 commented Jun 25, 2016 edited

This PR adds a provider for the scaleway ARM cloud hosting provider.

One can manage the entire scaleway stack like this:

provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}


resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

The PR includes tests & documentation.

Looking forward to your feedback!

@nicolai86 nicolai86 referenced this pull request in scaleway/scaleway-cli Jun 25, 2016
Merged

Introduce Logger #369

@nicolai86
Collaborator

Test output (cleaned up):


TF_ACC=1 go test ./builtin/providers/scaleway -v -run=TestAccScaleway -timeout 120m
=== RUN   TestAccScalewayIP_Basic
--- PASS: TestAccScalewayIP_Basic (1.96s)
=== RUN   TestAccScalewaySecurityGroupRule_Basic
--- PASS: TestAccScalewaySecurityGroupRule_Basic (2.77s)
=== RUN   TestAccScalewaySecurityGroup_Basic
--- PASS: TestAccScalewaySecurityGroup_Basic (1.94s)
=== RUN   TestAccScalewayServer_Basic
--- PASS: TestAccScalewayServer_Basic (123.19s)
=== RUN   TestAccScalewayVolumeAttachment_Basic
--- PASS: TestAccScalewayVolumeAttachment_Basic (325.58s)
=== RUN   TestAccScalewayVolume_Basic
--- PASS: TestAccScalewayVolume_Basic (2.61s)
PASS
ok    github.com/hashicorp/terraform/builtin/providers/scaleway  458.068s
@stack72 stack72 commented on the diff Jul 12, 2016
builtin/providers/scaleway/helpers.go
+
+ var currentState string
+
+ for {
+ server, err = s.GetServer(serverID)
+ if err != nil {
+ return err
+ }
+ if currentState != server.State {
+ fmt.Printf("Server changed state to %q\n", server.State)
+ currentState = server.State
+ }
+ if server.State == targetState {
+ break
+ }
+ time.Sleep(1 * time.Second)
@stack72
stack72 Jul 12, 2016 Member

is there any issue with 1 second refresh times? I.e. will we hit their API limit or anything?

@nicolai86
nicolai86 Jul 12, 2016 Collaborator

as far as I know there's no api limit. But maybe @moul can give some insight here.

@stack72 stack72 and 1 other commented on an outdated diff Jul 12, 2016
builtin/providers/scaleway/resource_scaleway_ip.go
+func resourceScalewayIPRead(d *schema.ResourceData, m interface{}) error {
+ scaleway := m.(*Client).scaleway
+ resp, err := scaleway.GetIP(d.Id())
+ if err != nil {
+ return err
+ }
+
+ d.Set("ip", resp.IP.Address)
+ d.Set("server", resp.IP.Server.Identifier)
+ return nil
+}
+
+func resourceScalewayIPUpdate(d *schema.ResourceData, m interface{}) error {
+ scaleway := m.(*Client).scaleway
+ if d.HasChange("server") {
+ if err := scaleway.AttachIP(d.Id(), d.Get("server").(string)); err != nil {
@stack72
stack72 Jul 12, 2016 Member

As server is Optional here, what happens if it is nil?

@nicolai86
nicolai86 Jul 12, 2016 Collaborator

It should detach the server. I'll add an explicit test case to make sure this behaves as expected. Sadly the scaleway cli does not expose a top level function like detachIP

@stack72 stack72 and 1 other commented on an outdated diff Jul 12, 2016
builtin/providers/scaleway/resource_scaleway_ip.go
+
+func resourceScalewayIPCreate(d *schema.ResourceData, m interface{}) error {
+ scaleway := m.(*Client).scaleway
+ resp, err := scaleway.NewIP()
+ if err != nil {
+ return err
+ }
+
+ d.SetId(resp.IP.ID)
+ return resourceScalewayIPUpdate(d, m)
+}
+
+func resourceScalewayIPRead(d *schema.ResourceData, m interface{}) error {
+ scaleway := m.(*Client).scaleway
+ resp, err := scaleway.GetIP(d.Id())
+ if err != nil {
@stack72
stack72 Jul 12, 2016 edited Member

Can we handle the issue when someone deletes an IP from the console / cli? We should do something like

if err != nil {
    if resp.StatusCode == 404 {
        d.SetId("")
        log.Printf("[DEBUG] IP Not Found so refreshing from state")
        return nil
    }
}

This is of course just pseudo code - i am not 100% sure of the actual API

@nicolai86
nicolai86 Jul 12, 2016 Collaborator

You can do this like this:

if serr, ok := err.(api.ScalewayAPIError); ok {
  if serr.StatusCode == 404 {
    d.SetId("")
    log.Printf("[DEBUG] IP Not Found when refreshing state")
    return nil
  }
}

Will be in the next commit.

@stack72 stack72 commented on an outdated diff Jul 12, 2016
...roviders/scaleway/resource_scaleway_security_group.go
+func resourceScalewaySecurityGroupUpdate(d *schema.ResourceData, m interface{}) error {
+ scaleway := m.(*Client).scaleway
+
+ var req = api.ScalewayNewSecurityGroup{
+ Organization: scaleway.Organization,
+ Name: d.Get("name").(string),
+ Description: d.Get("description").(string),
+ }
+
+ if err := scaleway.PutSecurityGroup(req, d.Id()); err != nil {
+ log.Printf("[DEBUG] Error reading security group: %q\n", err)
+
+ return err
+ }
+
+ return nil
@stack72
stack72 Jul 12, 2016 Member

I think we should call the Read func here - that way we can make sure that we set anything that changes back to state

@stack72 stack72 commented on an outdated diff Jul 12, 2016
...ers/scaleway/resource_scaleway_security_group_rule.go
+
+ var req = api.ScalewayNewSecurityGroupRule{
+ Action: d.Get("action").(string),
+ Direction: d.Get("direction").(string),
+ IPRange: d.Get("ip_range").(string),
+ Protocol: d.Get("protocol").(string),
+ DestPortFrom: d.Get("port").(int),
+ }
+
+ if err := scaleway.PutSecurityGroupRule(req, d.Get("security_group").(string), d.Id()); err != nil {
+ log.Printf("[DEBUG] error updating Security Group Rule: %q", err)
+
+ return err
+ }
+
+ return nil
@stack72
stack72 Jul 12, 2016 Member

Think we should call the Read func here

@stack72 stack72 commented on an outdated diff Jul 12, 2016
builtin/providers/scaleway/resource_scaleway_server.go
+ var req api.ScalewayServerPatchDefinition
+
+ if d.HasChange("name") {
+ name := d.Get("name").(string)
+ req.Name = &name
+ }
+
+ if d.HasChange("dynamic_ip_required") {
+ req.DynamicIPRequired = Bool(d.Get("dynamic_ip_required").(bool))
+ }
+
+ if err := scaleway.PatchServer(d.Id(), req); err != nil {
+ return fmt.Errorf("Failed patching scaleway server: %q", err)
+ }
+
+ return nil
@stack72
stack72 Jul 12, 2016 Member

Again call the Read func here

@stack72
Member
stack72 commented Jul 12, 2016

Hi @nicolai86

I left a small comments - nothing major. If we can get them fixed up then that will be fantastic.

I need to try and get in touch with Scaleway to see if we can get a testing account for terraform

Paul

@nicolai86
Collaborator

Hey @stack72

thank you for your feedback. Will adjust this soonish.

Best,
Raphael

@nicolai86
Collaborator
nicolai86 commented Jul 12, 2016 edited

@stack72 I think I adjusted everything based on your feedback. Ready for another review I'd say.
I've also rebased on current master to make sure we're in sync.

nicolai86 added some commits Jun 5, 2016
@nicolai86 nicolai86 Add scaleway provider
this PR allows the entire scaleway stack to be managed with terraform

example usage looks like this:

```
provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}
```

Note that volume attachments require the server to be stopped, which can lead to
downtimes of you attach new volumes to already used servers
044635e
@nicolai86 nicolai86 Update IP read to handle 404 gracefully dea5686
@nicolai86 nicolai86 Read back resource on update aedb826
@nicolai86 nicolai86 Ensure IP detachment works as expected
Sadly this is not part of the official scaleway api just yet
51d1882
@nicolai86 nicolai86 referenced this pull request in scaleway/scaleway-cli Jul 12, 2016
Merged

Add DetachIP helper #378

@stack72
Member
stack72 commented Jul 12, 2016

Hi @nicolai86

This looks good to me now. I just need to get an account on scale way to test this out and we are GTG

Paul

@stack72 stack72 commented on an outdated diff Jul 13, 2016
...te/source/docs/providers/scaleway/index.html.markdown
@@ -0,0 +1,96 @@
+---
+layout: "scaleway"
+page_title: "Provider: Scaleway"
+sidebar_current: "docs-scaleway-index"
+description: |-
+ The Docker provider is used to interact with Docker containers and images.
@stack72
stack72 Jul 13, 2016 Member

This description should be for Scaleway :)

@stack72 stack72 commented on an outdated diff Jul 13, 2016
...te/source/docs/providers/scaleway/index.html.markdown
@@ -0,0 +1,96 @@
+---
+layout: "scaleway"
+page_title: "Provider: Scaleway"
+sidebar_current: "docs-scaleway-index"
+description: |-
+ The Docker provider is used to interact with Docker containers and images.
+---
+
+# Scaleway Provider
+
+The Scaleway provider is used to manage Scaleway resources.
+
+Use the navigation to the left to read about the available resources.
+
+<div class="alert alert-block alert-info">
@stack72
stack72 Jul 13, 2016 Member

We can drop this block :)

@stack72 stack72 commented on an outdated diff Jul 13, 2016
...te/source/docs/providers/scaleway/index.html.markdown
+ dest_port_from = 80
+}
+
+resource "scaleway_security_group_rule" "https_accept" {
+ security_group = "${scaleway_security_group.http.id}"
+
+ action = "accept"
+ direction = "inbound"
+ ip_range = "0.0.0.0/0"
+ protocol = "TCP"
+ dest_port_from = 443
+}
+
+```
+
+You'll need to provide your Scaleway organization and Auth token,
@stack72
stack72 Jul 13, 2016 Member

So I got confused here on what an organisation key was. It took me a while to realise that it was actually access_key - I think we should rename the env var to that.

@stack72
Member
stack72 commented Jul 13, 2016

Hi @nicolai86

left 2 more small points on the docs side of things. Just managed to test this and the tests work as expected :)

The tests are very chatty though:

=== RUN   TestAccScalewaySecurityGroup_Basic
2016/07/13 10:46:30 POST /security_groups
2016/07/13 10:46:30 GET /security_groups
2016/07/13 10:46:30 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 DELETE /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
--- PASS: TestAccScalewaySecurityGroup_Basic (1.56s)

Is there any way that this can be disabled? I.e can they be moved within our logging levels of INFO, DEBUG, TRACE and ERROR etc?

P.

@stack72 stack72 self-assigned this Jul 13, 2016
@nicolai86
Collaborator

@stack72 I've updated the docs, renamed the api_key to cleary communicate the usage, and also made sure the logs are less chatty :)

@stack72
Member
stack72 commented Jul 13, 2016

Hi @nicolai86

This now looks great and the tests pass as expected. Thanks for making those logging changes!

Paul

@stack72 stack72 merged commit 9081cab into hashicorp:master Jul 13, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@nicolai86 nicolai86 deleted the nicolai86:feature/scaleway branch Jul 13, 2016
@nicolai86
Collaborator

Hey @stack72

thanks! If there are any bugs/ problems related to this feel free to pull me into the discussion.

Raphael

@stack72
Member
stack72 commented Jul 13, 2016

Will do! Thanks for all the help here :)

@iceycake iceycake added a commit to Ticketmaster/terraform that referenced this pull request Jul 22, 2016
@nicolai86 @iceycake nicolai86 + iceycake Add scaleway provider (#7331)
* Add scaleway provider

this PR allows the entire scaleway stack to be managed with terraform

example usage looks like this:

```
provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}
```

Note that volume attachments require the server to be stopped, which can lead to
downtimes of you attach new volumes to already used servers

* Update IP read to handle 404 gracefully

* Read back resource on update

* Ensure IP detachment works as expected

Sadly this is not part of the official scaleway api just yet

* Adjust detachIP helper

based on feedback from @QuentinPerez in
scaleway/scaleway-cli#378

* Cleanup documentation

* Rename api_key to access_key

following @stack72 suggestion and rename the provider api_key for more clarity

* Make tests less chatty by using custom logger
7095057
@sheerun
Contributor
sheerun commented Sep 5, 2016

@nicolai86 For some reason remote-exec provisioner is not working with scaleway resouces. It's stuck in endless loop:

scaleway_server.tunnel: Still creating... (3m20s elapsed)
scaleway_server.tunnel (remote-exec): Connecting to remote host via SSH...
scaleway_server.tunnel (remote-exec):   Host:
scaleway_server.tunnel (remote-exec):   User: root
scaleway_server.tunnel (remote-exec):   Password: false
scaleway_server.tunnel (remote-exec):   Private key: false
scaleway_server.tunnel (remote-exec):   SSH Agent: true
@nicolai86
Collaborator
nicolai86 commented Sep 5, 2016 edited

@sheerun I've got a more complex example of the scaleway provider up on github which works just fine with remote provisioner.

Assuming you can SSH into your instance this should work just fine. Note that this implies either a) dynamic_ip_required = true, because otherwise your instance does not have a public accessible ip, or b) the usage of a jump host.

The above repo demos both.

@nicolai86
Collaborator

To summarize the use of remote-exec:

option A
use of dynamic_ip_required

resource "scaleway_server" "server" {
  image               = "${var.image}"
  type                = "${var.type}"
  dynamic_ip_required = true

  provisioner "remote-exec" {
    inline = "echo hello world"
  }
}

this works because setting dynamic_ip_required = true gives us a public IP to connect to.

option B
assuming you have a jump host with a public accessible IP (see option A to create one)
use the jump host to connect:

resource "scaleway_server" "server" {
  image = "${var.image}"
  type  = "${var.type}"

  connection {
    type         = "ssh"
    user         = "root"
    host         = "${self.private_ip}"
    bastion_host = "${var.bastion_host}"
    bastion_user = "root"
    agent        = true
  }

  provisioner "remote-exec" {
    inline = "echo hello world"
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment