Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCE] Support for HTTP(S) proxies with BackendServices. #856

Closed
wants to merge 1 commit into from

Conversation

supertom
Copy link
Contributor

@supertom supertom commented Aug 31, 2016

[GCE] Support for HTTP(S) proxies with BackendServices.

Description

HTTP(S) proxies with Backend services are now supported. Users can now take advantage of creating a global forwarding rule that maps to a Managed Instance Group (via a Backend Service) with these changes.

High level flow of creating a forwarding rule:

be = gce.ex_create_backend(instance_group=ig)
bes = gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
                    description='bes desc', timeout_sec=60)

urlmap = gce.ex_create_urlmap('myurlmap', bes)

# create cert resources for use with https proxy
print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

proxy = gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

... create forwarding rule to point to to HTTPS proxy...

The output of demos/gce_demo.py illustrates this as well.

=> Backend Service w/Global Forwarding Rule demo/test start time: 2016-10-04 17:20:08.061679
=> Project: my-gcp-project
=> Create a BackendService
=>     InstanceTemplate "lct-instancetemplate" created
=>     Managed Instance Group [CENTRAL] "lct-mig-central" created
=>     ... MIG instances created: lct-mig-central-0h6c,lct-mig-central-tpzx
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Managed Instance Group [EAST] "lct-mig-east" created
=>     ... MIG instances created: lct-mig-east-0as7,lct-mig-east-gnoi
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Healthcheck lct-healthcheck created
=>     Backend Service "lct-bes" created
=>     URLMap "lct-urlmap" created
=>     TargetProxy "lct-httptargetproxy" created
=>     Address "lct-address" created with IP "107.178.255.112"
=>     Global Forwarding Rule "lct-gfr" created
=>     Firewall lct-firewall created
=> Cleaning up lct resources created
=>    Deleted lct-firewall (GCEFirewall)
=>    Deleted lct-gfr (GCEForwardingRule)
=>    Deleted lct-address (GCEAddress)
=>    Deleted lct-httptargetproxy (GCETargetHttpProxy)
=>    Deleted lct-urlmap (GCEUrlMap)
=>    Deleted lct-bes (GCEBackendService)
=>    Deleted lct-healthcheck (GCEHealthCheck)
=>    Deleted lct-mig-central (GCEInstanceGroupManager)
=>    Deleted lct-mig-east (GCEInstanceGroupManager)
=>    Deleted lct-instancetemplate (GCEInstanceTemplate)
=> Total runtime: 0:03:20.335315

Status

  • done, ready for review

Checklist (tick everything that applies)

  • Code linting (required, can be done after the PR checks)
  • Documentation
  • Tests
  • ICLA (required for bigger changes)

/cc @erjohnso

@@ -7,14 +7,14 @@
"backends": [
{
"description": "",
"group": "https://www.googleapis.com/resourceviews/v1beta1/projects/project_name/zones/us-central1-b/resourceViews/us-resources",
"group": "https://content.googleapis.com/compute/v1/projects/project_name/zones/us-central1-a/instanceGroups/myinstancegroup",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really? The URL has content vs www? That seems like something we should look investigate internally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, you're right. Should have been www. I corrected this link and others I saw that were not correct in the fixtures that I created lately, but I think a fixture review/regeneration is in order, as we've talked about.

@erjohnso
Copy link
Member

erjohnso commented Sep 1, 2016

For a new set of features like this, I usually try to include working examples in demos/gce_demo.py also. Doing that can serve (somewhat) as both documentation and a live integration test.

I typically run this script every time we make some change that has a risk of introducing a bug or breaking change. While it doesn't come close to full test cases or code coverage, it's a small safety net.

Do you mind extending the script for to include this new work?

@supertom
Copy link
Contributor Author

supertom commented Sep 1, 2016

It's a good idea, and in fact, it's how I test my changes.

The blocker/slowdown in doing it was not having 'create' support for InstanceTemplates or Unmanaged Instance Groups. As these are features that are not used often (at least, I suspect that they aren't), I had put them off. But, it's not too much effort so I think I will just add them in and then update the demo.

@supertom supertom force-pushed the gce-backend-support branch 4 times, most recently from 3ee7b39 to 34daae3 Compare October 3, 2016 21:32
@supertom
Copy link
Contributor Author

supertom commented Oct 3, 2016

Looks like the recent failures are due to deprecation warnings, and not (or no longer) caused by GCE. Is that correct? Someone mind having a look to confirm?

EDIT: nm, I didn't see that the issue travis flagged was actually my own. Fixed.

@supertom supertom force-pushed the gce-backend-support branch 2 times, most recently from 398d14e to 3a952fa Compare October 4, 2016 17:19
…ort GCEBackendService, GCESslCertificate, GCEBackend, and tests.

High level of what was added:

be = gce.ex_create_backend(instance_group=ig)
print gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
	                description='bes desc', timeout_sec=60)

... create URLMap that links to backend service ..

print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

print gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

Including modifications to backendservice as needed.  Now, with a forwarding rule, load balancer can be created that resolves to a Managed Instance Group via a Backend Service creation.

Unmanaged Instance Group support has also been added.  InstanceTemplate support (of note is the ability to create InstanceTemplates) has been added as well.

gce_demo now supports creating a backend service w/Global Forwarding by using the --backend-service flag.
@tonybaloney
Copy link
Contributor

@erjohnso @supertom where are we up to with this PR?

@erjohnso
Copy link
Member

Looks like there are some pep8 violations in the test fixtures that should be cleaned up at some point. The way we have tox tests set up, they're not being surfaced, but when I applied the patch, I saw this output:

Applying: Added support for HTTP(S) proxies with BackendServices. Includes support GCEBackendService, GCESslCertificate, GCEBackend, and tests.
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2762: trailing whitespace.
    "status": "PENDING", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2763: trailing whitespace.
    "kind": "compute#operation", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2764: trailing whitespace.
    "name": "my_instance_template1", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2765: trailing whitespace.
    "insertTime": "2016-09-02T09:31:52.285-07:00", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2766: trailing whitespace.
    "targetLink": "https://www.googleapis.com/compute/v1/projects/project_name/global/instanceTemplates/my_instance_template1", 
warning: squelched 113 whitespace errors
warning: 118 lines add whitespace errors.

These errors are not super helpful in tracking them down, but I did confirm that there are trailing spaces in https://github.com/supertom/libcloud/blob/c332b347cc92bb1e67c02ff6e03d2620a09e68d4/libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json

Poking around a bit more, looks like it's been this way for a while in a few other spots.

$ find libcloud/test -name '*.json' -exec grep -l '.*\s$' {} \;
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_get.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_add.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_edit.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_list.json
libcloud/test/loadbalancer/fixtures/gogrid/ip_list.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_3xxx.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes.json
libcloud/test/dns/fixtures/auroradns/zone_example_com_records.json
libcloud/test/dns/fixtures/auroradns/zone_list.json
libcloud/test/container/fixtures/ecs/stoptask.json
libcloud/test/compute/fixtures/rimuhosting/r_orders.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps_running_state.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_new_vps.json
libcloud/test/compute/fixtures/rimuhosting/r_pricing_plans.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/elastichosts/drives_info.json
libcloud/test/compute/fixtures/elastichosts/drives_create.json
libcloud/test/compute/fixtures/elastichosts/servers_create.json
libcloud/test/compute/fixtures/openstack_identity/v3/v3_users_a.json
libcloud/test/compute/fixtures/gogrid/lookup_list_ip_datacenter.json
libcloud/test/compute/fixtures/gogrid/password_list.json
libcloud/test/compute/fixtures/gogrid/image_save.json
libcloud/test/compute/fixtures/gogrid/ip_list.json
libcloud/test/compute/fixtures/ecp/vm_list.json
libcloud/test/compute/fixtures/ecp/vm_1_action_stop.json
libcloud/test/compute/fixtures/ecp/vm_1_get.json
libcloud/test/compute/fixtures/ecp/vm_1_action_start.json
libcloud/test/compute/fixtures/ecp/htemplate_list.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_group_rules_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_servers_1c01300f-ef97-4937-8f03-ac676d6234be_os-security-groups.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_a.json
libcloud/test/compute/fixtures/brightbox/list_images.json
libcloud/test/compute/fixtures/brightbox/create_cloud_ip.json
libcloud/test/compute/fixtures/brightbox/list_zones.json
libcloud/test/compute/fixtures/brightbox/list_server_types.json
libcloud/test/compute/fixtures/brightbox/list_servers.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_b.json
libcloud/test/compute/fixtures/brightbox/list_cloud_ips.json

I also upgraded packages on my dev machine, so it's possible this is new output (to me) that wasn't surfaced from my old dev packages.

@asfgit asfgit closed this in cffd964 Oct 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants