New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCE] Support for HTTP(S) proxies with BackendServices. #856

Closed
wants to merge 1 commit into
base: trunk
from

Conversation

Projects
None yet
3 participants
@supertom
Contributor

supertom commented Aug 31, 2016

[GCE] Support for HTTP(S) proxies with BackendServices.

Description

HTTP(S) proxies with Backend services are now supported. Users can now take advantage of creating a global forwarding rule that maps to a Managed Instance Group (via a Backend Service) with these changes.

High level flow of creating a forwarding rule:

be = gce.ex_create_backend(instance_group=ig)
bes = gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
                    description='bes desc', timeout_sec=60)

urlmap = gce.ex_create_urlmap('myurlmap', bes)

# create cert resources for use with https proxy
print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

proxy = gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

... create forwarding rule to point to to HTTPS proxy...

The output of demos/gce_demo.py illustrates this as well.

=> Backend Service w/Global Forwarding Rule demo/test start time: 2016-10-04 17:20:08.061679
=> Project: my-gcp-project
=> Create a BackendService
=>     InstanceTemplate "lct-instancetemplate" created
=>     Managed Instance Group [CENTRAL] "lct-mig-central" created
=>     ... MIG instances created: lct-mig-central-0h6c,lct-mig-central-tpzx
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Managed Instance Group [EAST] "lct-mig-east" created
=>     ... MIG instances created: lct-mig-east-0as7,lct-mig-east-gnoi
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Healthcheck lct-healthcheck created
=>     Backend Service "lct-bes" created
=>     URLMap "lct-urlmap" created
=>     TargetProxy "lct-httptargetproxy" created
=>     Address "lct-address" created with IP "107.178.255.112"
=>     Global Forwarding Rule "lct-gfr" created
=>     Firewall lct-firewall created
=> Cleaning up lct resources created
=>    Deleted lct-firewall (GCEFirewall)
=>    Deleted lct-gfr (GCEForwardingRule)
=>    Deleted lct-address (GCEAddress)
=>    Deleted lct-httptargetproxy (GCETargetHttpProxy)
=>    Deleted lct-urlmap (GCEUrlMap)
=>    Deleted lct-bes (GCEBackendService)
=>    Deleted lct-healthcheck (GCEHealthCheck)
=>    Deleted lct-mig-central (GCEInstanceGroupManager)
=>    Deleted lct-mig-east (GCEInstanceGroupManager)
=>    Deleted lct-instancetemplate (GCEInstanceTemplate)
=> Total runtime: 0:03:20.335315

Status

  • done, ready for review

Checklist (tick everything that applies)

  • Code linting (required, can be done after the PR checks)
  • Documentation
  • Tests
  • ICLA (required for bigger changes)

/cc @erjohnso

Show outdated Hide outdated libcloud/test/compute/fixtures/gce/global_backendServices_web_service.json
@@ -7,14 +7,14 @@
"backends": [
{
"description": "",
"group": "https://www.googleapis.com/resourceviews/v1beta1/projects/project_name/zones/us-central1-b/resourceViews/us-resources",
"group": "https://content.googleapis.com/compute/v1/projects/project_name/zones/us-central1-a/instanceGroups/myinstancegroup",

This comment has been minimized.

@erjohnso

erjohnso Sep 1, 2016

Member

Really? The URL has content vs www? That seems like something we should look investigate internally.

@erjohnso

erjohnso Sep 1, 2016

Member

Really? The URL has content vs www? That seems like something we should look investigate internally.

This comment has been minimized.

@supertom

supertom Sep 1, 2016

Contributor

Nope, you're right. Should have been www. I corrected this link and others I saw that were not correct in the fixtures that I created lately, but I think a fixture review/regeneration is in order, as we've talked about.

@supertom

supertom Sep 1, 2016

Contributor

Nope, you're right. Should have been www. I corrected this link and others I saw that were not correct in the fixtures that I created lately, but I think a fixture review/regeneration is in order, as we've talked about.

@erjohnso

This comment has been minimized.

Show comment
Hide comment
@erjohnso

erjohnso Sep 1, 2016

Member

For a new set of features like this, I usually try to include working examples in demos/gce_demo.py also. Doing that can serve (somewhat) as both documentation and a live integration test.

I typically run this script every time we make some change that has a risk of introducing a bug or breaking change. While it doesn't come close to full test cases or code coverage, it's a small safety net.

Do you mind extending the script for to include this new work?

Member

erjohnso commented Sep 1, 2016

For a new set of features like this, I usually try to include working examples in demos/gce_demo.py also. Doing that can serve (somewhat) as both documentation and a live integration test.

I typically run this script every time we make some change that has a risk of introducing a bug or breaking change. While it doesn't come close to full test cases or code coverage, it's a small safety net.

Do you mind extending the script for to include this new work?

@supertom

This comment has been minimized.

Show comment
Hide comment
@supertom

supertom Sep 1, 2016

Contributor

It's a good idea, and in fact, it's how I test my changes.

The blocker/slowdown in doing it was not having 'create' support for InstanceTemplates or Unmanaged Instance Groups. As these are features that are not used often (at least, I suspect that they aren't), I had put them off. But, it's not too much effort so I think I will just add them in and then update the demo.

Contributor

supertom commented Sep 1, 2016

It's a good idea, and in fact, it's how I test my changes.

The blocker/slowdown in doing it was not having 'create' support for InstanceTemplates or Unmanaged Instance Groups. As these are features that are not used often (at least, I suspect that they aren't), I had put them off. But, it's not too much effort so I think I will just add them in and then update the demo.

@supertom

This comment has been minimized.

Show comment
Hide comment
@supertom

supertom Oct 3, 2016

Contributor

Looks like the recent failures are due to deprecation warnings, and not (or no longer) caused by GCE. Is that correct? Someone mind having a look to confirm?

EDIT: nm, I didn't see that the issue travis flagged was actually my own. Fixed.

Contributor

supertom commented Oct 3, 2016

Looks like the recent failures are due to deprecation warnings, and not (or no longer) caused by GCE. Is that correct? Someone mind having a look to confirm?

EDIT: nm, I didn't see that the issue travis flagged was actually my own. Fixed.

Added support for HTTP(S) proxies with BackendServices. Includes supp…
…ort GCEBackendService, GCESslCertificate, GCEBackend, and tests.

High level of what was added:

be = gce.ex_create_backend(instance_group=ig)
print gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
	                description='bes desc', timeout_sec=60)

... create URLMap that links to backend service ..

print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

print gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

Including modifications to backendservice as needed.  Now, with a forwarding rule, load balancer can be created that resolves to a Managed Instance Group via a Backend Service creation.

Unmanaged Instance Group support has also been added.  InstanceTemplate support (of note is the ability to create InstanceTemplates) has been added as well.

gce_demo now supports creating a backend service w/Global Forwarding by using the --backend-service flag.
@tonybaloney

This comment has been minimized.

Show comment
Hide comment
@tonybaloney

tonybaloney Oct 11, 2016

Contributor

@erjohnso @supertom where are we up to with this PR?

Contributor

tonybaloney commented Oct 11, 2016

@erjohnso @supertom where are we up to with this PR?

@erjohnso

This comment has been minimized.

Show comment
Hide comment
@erjohnso

erjohnso Oct 13, 2016

Member

Looks like there are some pep8 violations in the test fixtures that should be cleaned up at some point. The way we have tox tests set up, they're not being surfaced, but when I applied the patch, I saw this output:

Applying: Added support for HTTP(S) proxies with BackendServices. Includes support GCEBackendService, GCESslCertificate, GCEBackend, and tests.
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2762: trailing whitespace.
    "status": "PENDING", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2763: trailing whitespace.
    "kind": "compute#operation", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2764: trailing whitespace.
    "name": "my_instance_template1", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2765: trailing whitespace.
    "insertTime": "2016-09-02T09:31:52.285-07:00", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2766: trailing whitespace.
    "targetLink": "https://www.googleapis.com/compute/v1/projects/project_name/global/instanceTemplates/my_instance_template1", 
warning: squelched 113 whitespace errors
warning: 118 lines add whitespace errors.

These errors are not super helpful in tracking them down, but I did confirm that there are trailing spaces in https://github.com/supertom/libcloud/blob/c332b347cc92bb1e67c02ff6e03d2620a09e68d4/libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json

Poking around a bit more, looks like it's been this way for a while in a few other spots.

$ find libcloud/test -name '*.json' -exec grep -l '.*\s$' {} \;
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_get.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_add.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_edit.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_list.json
libcloud/test/loadbalancer/fixtures/gogrid/ip_list.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_3xxx.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes.json
libcloud/test/dns/fixtures/auroradns/zone_example_com_records.json
libcloud/test/dns/fixtures/auroradns/zone_list.json
libcloud/test/container/fixtures/ecs/stoptask.json
libcloud/test/compute/fixtures/rimuhosting/r_orders.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps_running_state.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_new_vps.json
libcloud/test/compute/fixtures/rimuhosting/r_pricing_plans.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/elastichosts/drives_info.json
libcloud/test/compute/fixtures/elastichosts/drives_create.json
libcloud/test/compute/fixtures/elastichosts/servers_create.json
libcloud/test/compute/fixtures/openstack_identity/v3/v3_users_a.json
libcloud/test/compute/fixtures/gogrid/lookup_list_ip_datacenter.json
libcloud/test/compute/fixtures/gogrid/password_list.json
libcloud/test/compute/fixtures/gogrid/image_save.json
libcloud/test/compute/fixtures/gogrid/ip_list.json
libcloud/test/compute/fixtures/ecp/vm_list.json
libcloud/test/compute/fixtures/ecp/vm_1_action_stop.json
libcloud/test/compute/fixtures/ecp/vm_1_get.json
libcloud/test/compute/fixtures/ecp/vm_1_action_start.json
libcloud/test/compute/fixtures/ecp/htemplate_list.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_group_rules_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_servers_1c01300f-ef97-4937-8f03-ac676d6234be_os-security-groups.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_a.json
libcloud/test/compute/fixtures/brightbox/list_images.json
libcloud/test/compute/fixtures/brightbox/create_cloud_ip.json
libcloud/test/compute/fixtures/brightbox/list_zones.json
libcloud/test/compute/fixtures/brightbox/list_server_types.json
libcloud/test/compute/fixtures/brightbox/list_servers.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_b.json
libcloud/test/compute/fixtures/brightbox/list_cloud_ips.json

I also upgraded packages on my dev machine, so it's possible this is new output (to me) that wasn't surfaced from my old dev packages.

Member

erjohnso commented Oct 13, 2016

Looks like there are some pep8 violations in the test fixtures that should be cleaned up at some point. The way we have tox tests set up, they're not being surfaced, but when I applied the patch, I saw this output:

Applying: Added support for HTTP(S) proxies with BackendServices. Includes support GCEBackendService, GCESslCertificate, GCEBackend, and tests.
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2762: trailing whitespace.
    "status": "PENDING", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2763: trailing whitespace.
    "kind": "compute#operation", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2764: trailing whitespace.
    "name": "my_instance_template1", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2765: trailing whitespace.
    "insertTime": "2016-09-02T09:31:52.285-07:00", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2766: trailing whitespace.
    "targetLink": "https://www.googleapis.com/compute/v1/projects/project_name/global/instanceTemplates/my_instance_template1", 
warning: squelched 113 whitespace errors
warning: 118 lines add whitespace errors.

These errors are not super helpful in tracking them down, but I did confirm that there are trailing spaces in https://github.com/supertom/libcloud/blob/c332b347cc92bb1e67c02ff6e03d2620a09e68d4/libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json

Poking around a bit more, looks like it's been this way for a while in a few other spots.

$ find libcloud/test -name '*.json' -exec grep -l '.*\s$' {} \;
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_get.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_add.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_edit.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_list.json
libcloud/test/loadbalancer/fixtures/gogrid/ip_list.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_3xxx.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes.json
libcloud/test/dns/fixtures/auroradns/zone_example_com_records.json
libcloud/test/dns/fixtures/auroradns/zone_list.json
libcloud/test/container/fixtures/ecs/stoptask.json
libcloud/test/compute/fixtures/rimuhosting/r_orders.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps_running_state.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_new_vps.json
libcloud/test/compute/fixtures/rimuhosting/r_pricing_plans.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/elastichosts/drives_info.json
libcloud/test/compute/fixtures/elastichosts/drives_create.json
libcloud/test/compute/fixtures/elastichosts/servers_create.json
libcloud/test/compute/fixtures/openstack_identity/v3/v3_users_a.json
libcloud/test/compute/fixtures/gogrid/lookup_list_ip_datacenter.json
libcloud/test/compute/fixtures/gogrid/password_list.json
libcloud/test/compute/fixtures/gogrid/image_save.json
libcloud/test/compute/fixtures/gogrid/ip_list.json
libcloud/test/compute/fixtures/ecp/vm_list.json
libcloud/test/compute/fixtures/ecp/vm_1_action_stop.json
libcloud/test/compute/fixtures/ecp/vm_1_get.json
libcloud/test/compute/fixtures/ecp/vm_1_action_start.json
libcloud/test/compute/fixtures/ecp/htemplate_list.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_group_rules_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_servers_1c01300f-ef97-4937-8f03-ac676d6234be_os-security-groups.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_a.json
libcloud/test/compute/fixtures/brightbox/list_images.json
libcloud/test/compute/fixtures/brightbox/create_cloud_ip.json
libcloud/test/compute/fixtures/brightbox/list_zones.json
libcloud/test/compute/fixtures/brightbox/list_server_types.json
libcloud/test/compute/fixtures/brightbox/list_servers.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_b.json
libcloud/test/compute/fixtures/brightbox/list_cloud_ips.json

I also upgraded packages on my dev machine, so it's possible this is new output (to me) that wasn't surfaced from my old dev packages.

@asfgit asfgit closed this in cffd964 Oct 13, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment