New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scaleway provider #7331

Merged
merged 8 commits into from Jul 13, 2016

Conversation

Projects
None yet
3 participants
@nicolai86
Copy link
Contributor

nicolai86 commented Jun 25, 2016

This PR adds a provider for the scaleway ARM cloud hosting provider.

One can manage the entire scaleway stack like this:

provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}


resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

The PR includes tests & documentation.

Looking forward to your feedback!

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch from 9c6d13e to 292f17e Jun 25, 2016

@nicolai86 nicolai86 referenced this pull request Jun 25, 2016

Merged

Introduce Logger #369

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch 11 times, most recently from 67e6dde to 7b41d31 Jun 25, 2016

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch 4 times, most recently from a4bcefe to cc559e8 Jun 26, 2016

@nicolai86

This comment has been minimized.

Copy link
Contributor

nicolai86 commented Jun 26, 2016

Test output (cleaned up):


TF_ACC=1 go test ./builtin/providers/scaleway -v -run=TestAccScaleway -timeout 120m
=== RUN   TestAccScalewayIP_Basic
--- PASS: TestAccScalewayIP_Basic (1.96s)
=== RUN   TestAccScalewaySecurityGroupRule_Basic
--- PASS: TestAccScalewaySecurityGroupRule_Basic (2.77s)
=== RUN   TestAccScalewaySecurityGroup_Basic
--- PASS: TestAccScalewaySecurityGroup_Basic (1.94s)
=== RUN   TestAccScalewayServer_Basic
--- PASS: TestAccScalewayServer_Basic (123.19s)
=== RUN   TestAccScalewayVolumeAttachment_Basic
--- PASS: TestAccScalewayVolumeAttachment_Basic (325.58s)
=== RUN   TestAccScalewayVolume_Basic
--- PASS: TestAccScalewayVolume_Basic (2.61s)
PASS
ok    github.com/hashicorp/terraform/builtin/providers/scaleway  458.068s

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch 2 times, most recently from 6b45a2d to 743f714 Jun 26, 2016

if server.State == targetState {
break
}
time.Sleep(1 * time.Second)

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

is there any issue with 1 second refresh times? I.e. will we hit their API limit or anything?

This comment has been minimized.

@nicolai86

nicolai86 Jul 12, 2016

Contributor

as far as I know there's no api limit. But maybe @moul can give some insight here.

func resourceScalewayIPUpdate(d *schema.ResourceData, m interface{}) error {
scaleway := m.(*Client).scaleway
if d.HasChange("server") {
if err := scaleway.AttachIP(d.Id(), d.Get("server").(string)); err != nil {

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

As server is Optional here, what happens if it is nil?

This comment has been minimized.

@nicolai86

nicolai86 Jul 12, 2016

Contributor

It should detach the server. I'll add an explicit test case to make sure this behaves as expected. Sadly the scaleway cli does not expose a top level function like detachIP

func resourceScalewayIPRead(d *schema.ResourceData, m interface{}) error {
scaleway := m.(*Client).scaleway
resp, err := scaleway.GetIP(d.Id())
if err != nil {

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

Can we handle the issue when someone deletes an IP from the console / cli? We should do something like

if err != nil {
    if resp.StatusCode == 404 {
        d.SetId("")
        log.Printf("[DEBUG] IP Not Found so refreshing from state")
        return nil
    }
}

This is of course just pseudo code - i am not 100% sure of the actual API

This comment has been minimized.

@nicolai86

nicolai86 Jul 12, 2016

Contributor

You can do this like this:

if serr, ok := err.(api.ScalewayAPIError); ok {
  if serr.StatusCode == 404 {
    d.SetId("")
    log.Printf("[DEBUG] IP Not Found when refreshing state")
    return nil
  }
}

Will be in the next commit.

return err
}

return nil

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

I think we should call the Read func here - that way we can make sure that we set anything that changes back to state

return err
}

return nil

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

Think we should call the Read func here

return fmt.Errorf("Failed patching scaleway server: %q", err)
}

return nil

This comment has been minimized.

@stack72

stack72 Jul 12, 2016

Contributor

Again call the Read func here

@stack72

This comment has been minimized.

Copy link
Contributor

stack72 commented Jul 12, 2016

Hi @nicolai86

I left a small comments - nothing major. If we can get them fixed up then that will be fantastic.

I need to try and get in touch with Scaleway to see if we can get a testing account for terraform

Paul

nicolai86 added some commits Jun 5, 2016

Add scaleway provider
this PR allows the entire scaleway stack to be managed with terraform

example usage looks like this:

```
provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}
```

Note that volume attachments require the server to be stopped, which can lead to
downtimes of you attach new volumes to already used servers

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch 2 times, most recently from fbac518 to 3be3cb2 Jul 12, 2016

Ensure IP detachment works as expected
Sadly this is not part of the official scaleway api just yet

@nicolai86 nicolai86 force-pushed the nicolai86:feature/scaleway branch from 3be3cb2 to 51d1882 Jul 12, 2016

@nicolai86 nicolai86 referenced this pull request Jul 12, 2016

Merged

Add DetachIP helper #378

@stack72

This comment has been minimized.

Copy link
Contributor

stack72 commented Jul 12, 2016

Hi @nicolai86

This looks good to me now. I just need to get an account on scale way to test this out and we are GTG

Paul

page_title: "Provider: Scaleway"
sidebar_current: "docs-scaleway-index"
description: |-
The Docker provider is used to interact with Docker containers and images.

This comment has been minimized.

@stack72

stack72 Jul 13, 2016

Contributor

This description should be for Scaleway :)


Use the navigation to the left to read about the available resources.

<div class="alert alert-block alert-info">

This comment has been minimized.

@stack72

stack72 Jul 13, 2016

Contributor

We can drop this block :)

```

You'll need to provide your Scaleway organization and Auth token,

This comment has been minimized.

@stack72

stack72 Jul 13, 2016

Contributor

So I got confused here on what an organisation key was. It took me a while to realise that it was actually access_key - I think we should rename the env var to that.

@stack72

This comment has been minimized.

Copy link
Contributor

stack72 commented Jul 13, 2016

Hi @nicolai86

left 2 more small points on the docs side of things. Just managed to test this and the tests work as expected :)

The tests are very chatty though:

=== RUN   TestAccScalewaySecurityGroup_Basic
2016/07/13 10:46:30 POST /security_groups
2016/07/13 10:46:30 GET /security_groups
2016/07/13 10:46:30 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 GET /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
2016/07/13 10:46:31 DELETE /security_groups/4d25daae-04f8-4dd6-b778-e3d234dafaf0
--- PASS: TestAccScalewaySecurityGroup_Basic (1.56s)

Is there any way that this can be disabled? I.e can they be moved within our logging levels of INFO, DEBUG, TRACE and ERROR etc?

P.

@stack72 stack72 self-assigned this Jul 13, 2016

nicolai86 added some commits Jul 13, 2016

Rename api_key to access_key
following @stack72 suggestion and rename the provider api_key for more clarity
@nicolai86

This comment has been minimized.

Copy link
Contributor

nicolai86 commented Jul 13, 2016

@stack72 I've updated the docs, renamed the api_key to cleary communicate the usage, and also made sure the logs are less chatty :)

@stack72

This comment has been minimized.

Copy link
Contributor

stack72 commented Jul 13, 2016

Hi @nicolai86

This now looks great and the tests pass as expected. Thanks for making those logging changes!

Paul

@stack72 stack72 merged commit 9081cab into hashicorp:master Jul 13, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@nicolai86 nicolai86 deleted the nicolai86:feature/scaleway branch Jul 13, 2016

@nicolai86

This comment has been minimized.

Copy link
Contributor

nicolai86 commented Jul 13, 2016

Hey @stack72

thanks! If there are any bugs/ problems related to this feel free to pull me into the discussion.

Raphael

@stack72

This comment has been minimized.

Copy link
Contributor

stack72 commented Jul 13, 2016

Will do! Thanks for all the help here :)

iceycake added a commit to ticketmaster/terraform that referenced this pull request Jul 22, 2016

Add scaleway provider (hashicorp#7331)
* Add scaleway provider

this PR allows the entire scaleway stack to be managed with terraform

example usage looks like this:

```
provider "scaleway" {
  api_key = "snap"
  organization = "snip"
}

resource "scaleway_ip" "base" {
  server = "${scaleway_server.base.id}"
}

resource "scaleway_server" "base" {
  name = "test"
  # ubuntu 14.04
  image = "aecaed73-51a5-4439-a127-6d8229847145"
  type = "C2S"
}

resource "scaleway_volume" "test" {
  name = "test"
  size_in_gb = 20
  type = "l_ssd"
}

resource "scaleway_volume_attachment" "test" {
  server = "${scaleway_server.base.id}"
  volume = "${scaleway_volume.test.id}"
}

resource "scaleway_security_group" "base" {
  name = "public"
  description = "public gateway"
}

resource "scaleway_security_group_rule" "http-ingress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "inbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}

resource "scaleway_security_group_rule" "http-egress" {
  security_group = "${scaleway_security_group.base.id}"

  action = "accept"
  direction = "outbound"
  ip_range = "0.0.0.0/0"
  protocol = "TCP"
  port = 80
}
```

Note that volume attachments require the server to be stopped, which can lead to
downtimes of you attach new volumes to already used servers

* Update IP read to handle 404 gracefully

* Read back resource on update

* Ensure IP detachment works as expected

Sadly this is not part of the official scaleway api just yet

* Adjust detachIP helper

based on feedback from @QuentinPerez in
scaleway/scaleway-cli#378

* Cleanup documentation

* Rename api_key to access_key

following @stack72 suggestion and rename the provider api_key for more clarity

* Make tests less chatty by using custom logger
@sheerun

This comment has been minimized.

Copy link
Contributor

sheerun commented Sep 5, 2016

@nicolai86 For some reason remote-exec provisioner is not working with scaleway resouces. It's stuck in endless loop:

scaleway_server.tunnel: Still creating... (3m20s elapsed)
scaleway_server.tunnel (remote-exec): Connecting to remote host via SSH...
scaleway_server.tunnel (remote-exec):   Host:
scaleway_server.tunnel (remote-exec):   User: root
scaleway_server.tunnel (remote-exec):   Password: false
scaleway_server.tunnel (remote-exec):   Private key: false
scaleway_server.tunnel (remote-exec):   SSH Agent: true
@nicolai86

This comment has been minimized.

Copy link
Contributor

nicolai86 commented Sep 5, 2016

@sheerun I've got a more complex example of the scaleway provider up on github which works just fine with remote provisioner.

Assuming you can SSH into your instance this should work just fine. Note that this implies either a) dynamic_ip_required = true, because otherwise your instance does not have a public accessible ip, or b) the usage of a jump host.

The above repo demos both.

@nicolai86

This comment has been minimized.

Copy link
Contributor

nicolai86 commented Sep 5, 2016

To summarize the use of remote-exec:

option A
use of dynamic_ip_required

resource "scaleway_server" "server" {
  image               = "${var.image}"
  type                = "${var.type}"
  dynamic_ip_required = true

  provisioner "remote-exec" {
    inline = "echo hello world"
  }
}

this works because setting dynamic_ip_required = true gives us a public IP to connect to.

option B
assuming you have a jump host with a public accessible IP (see option A to create one)
use the jump host to connect:

resource "scaleway_server" "server" {
  image = "${var.image}"
  type  = "${var.type}"

  connection {
    type         = "ssh"
    user         = "root"
    host         = "${self.private_ip}"
    bastion_host = "${var.bastion_host}"
    bastion_user = "root"
    agent        = true
  }

  provisioner "remote-exec" {
    inline = "echo hello world"
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment