Skip to content

[GCE] Support for HTTP(S) proxies with BackendServices.#856

Closed
supertom wants to merge 1 commit into
apache:trunkfrom
supertom:gce-backend-support
Closed

[GCE] Support for HTTP(S) proxies with BackendServices.#856
supertom wants to merge 1 commit into
apache:trunkfrom
supertom:gce-backend-support

Conversation

@supertom
Copy link
Copy Markdown
Contributor

@supertom supertom commented Aug 31, 2016 •
edited
Loading

[GCE] Support for HTTP(S) proxies with BackendServices.

Description

HTTP(S) proxies with Backend services are now supported. Users can now take advantage of creating a global forwarding rule that maps to a Managed Instance Group (via a Backend Service) with these changes.

High level flow of creating a forwarding rule:

be = gce.ex_create_backend(instance_group=ig)
bes = gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
                    description='bes desc', timeout_sec=60)

urlmap = gce.ex_create_urlmap('myurlmap', bes)

# create cert resources for use with https proxy
print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

proxy = gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

... create forwarding rule to point to to HTTPS proxy...

The output of demos/gce_demo.py illustrates this as well.

=> Backend Service w/Global Forwarding Rule demo/test start time: 2016-10-04 17:20:08.061679
=> Project: my-gcp-project
=> Create a BackendService
=>     InstanceTemplate "lct-instancetemplate" created
=>     Managed Instance Group [CENTRAL] "lct-mig-central" created
=>     ... MIG instances created: lct-mig-central-0h6c,lct-mig-central-tpzx
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Managed Instance Group [EAST] "lct-mig-east" created
=>     ... MIG instances created: lct-mig-east-0as7,lct-mig-east-gnoi
=>     ... MIG ports set: [{'name': 'lct-http', 'port': 80}]
=>     Healthcheck lct-healthcheck created
=>     Backend Service "lct-bes" created
=>     URLMap "lct-urlmap" created
=>     TargetProxy "lct-httptargetproxy" created
=>     Address "lct-address" created with IP "107.178.255.112"
=>     Global Forwarding Rule "lct-gfr" created
=>     Firewall lct-firewall created
=> Cleaning up lct resources created
=>    Deleted lct-firewall (GCEFirewall)
=>    Deleted lct-gfr (GCEForwardingRule)
=>    Deleted lct-address (GCEAddress)
=>    Deleted lct-httptargetproxy (GCETargetHttpProxy)
=>    Deleted lct-urlmap (GCEUrlMap)
=>    Deleted lct-bes (GCEBackendService)
=>    Deleted lct-healthcheck (GCEHealthCheck)
=>    Deleted lct-mig-central (GCEInstanceGroupManager)
=>    Deleted lct-mig-east (GCEInstanceGroupManager)
=>    Deleted lct-instancetemplate (GCEInstanceTemplate)
=> Total runtime: 0:03:20.335315

Status

  • done, ready for review

Checklist (tick everything that applies)

  • Code linting (required, can be done after the PR checks)
  • Documentation
  • Tests
  • ICLA (required for bigger changes)

/cc @erjohnso

erjohnso
erjohnso reviewed Sep 1, 2016
View reviewed changes
Comment thread libcloud/test/compute/fixtures/gce/global_backendServices_web_service.json Outdated
{
"description": "",
"group": "https://www.googleapis.com/resourceviews/v1beta1/projects/project_name/zones/us-central1-b/resourceViews/us-resources",
"group": "https://content.googleapis.com/compute/v1/projects/project_name/zones/us-central1-a/instanceGroups/myinstancegroup",
Copy link
Copy Markdown
Contributor

@erjohnso erjohnso Sep 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really? The URL has content vs www? That seems like something we should look investigate internally.

Copy link
Copy Markdown
Contributor Author

@supertom supertom Sep 1, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, you're right. Should have been www. I corrected this link and others I saw that were not correct in the fixtures that I created lately, but I think a fixture review/regeneration is in order, as we've talked about.

@erjohnso
Copy link
Copy Markdown
Contributor

erjohnso commented Sep 1, 2016

For a new set of features like this, I usually try to include working examples in demos/gce_demo.py also. Doing that can serve (somewhat) as both documentation and a live integration test.

I typically run this script every time we make some change that has a risk of introducing a bug or breaking change. While it doesn't come close to full test cases or code coverage, it's a small safety net.

Do you mind extending the script for to include this new work?

@supertom
Copy link
Copy Markdown
Contributor Author

supertom commented Sep 1, 2016

It's a good idea, and in fact, it's how I test my changes.

The blocker/slowdown in doing it was not having 'create' support for InstanceTemplates or Unmanaged Instance Groups. As these are features that are not used often (at least, I suspect that they aren't), I had put them off. But, it's not too much effort so I think I will just add them in and then update the demo.

@supertom supertom force-pushed the gce-backend-support branch 4 times, most recently from 3ee7b39 to 34daae3 Compare October 3, 2016 21:32
@supertom
Copy link
Copy Markdown
Contributor Author

supertom commented Oct 3, 2016 •
edited
Loading

Looks like the recent failures are due to deprecation warnings, and not (or no longer) caused by GCE. Is that correct? Someone mind having a look to confirm?

EDIT: nm, I didn't see that the issue travis flagged was actually my own. Fixed.

@supertom supertom force-pushed the gce-backend-support branch 2 times, most recently from 398d14e to 3a952fa Compare October 4, 2016 17:19
@supertom
Added support for HTTP(S) proxies with BackendServices. Includes supp…
c332b34 
…ort GCEBackendService, GCESslCertificate, GCEBackend, and tests.

High level of what was added:

be = gce.ex_create_backend(instance_group=ig)
print gce.ex_create_backendservice(
            bes_name, [hc], backends=[be], port_name='my-ssl', protocol='HTTP',
	                description='bes desc', timeout_sec=60)

... create URLMap that links to backend service ..

print gce.ex_create_sslcertificate(ssl_name, certificate=certificate, private_key=private_key, description=ssl_desc)
print gce.ex_list_sslcertificates()

print gce.ex_create_targethttpsproxy(targethttpproxy_name, urlmap, ssl_list)
print gce.ex_list_targethttpsproxies()

Including modifications to backendservice as needed.  Now, with a forwarding rule, load balancer can be created that resolves to a Managed Instance Group via a Backend Service creation.

Unmanaged Instance Group support has also been added.  InstanceTemplate support (of note is the ability to create InstanceTemplates) has been added as well.

gce_demo now supports creating a backend service w/Global Forwarding by using the --backend-service flag.
@supertom supertom force-pushed the gce-backend-support branch from 3a952fa to c332b34 Compare October 4, 2016 20:13
@tonybaloney
Copy link
Copy Markdown
Contributor

tonybaloney commented Oct 11, 2016

@erjohnso @supertom where are we up to with this PR?

@erjohnso
Copy link
Copy Markdown
Contributor

erjohnso commented Oct 13, 2016

Looks like there are some pep8 violations in the test fixtures that should be cleaned up at some point. The way we have tox tests set up, they're not being surfaced, but when I applied the patch, I saw this output:

Applying: Added support for HTTP(S) proxies with BackendServices. Includes support GCEBackendService, GCESslCertificate, GCEBackend, and tests.
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2762: trailing whitespace.
    "status": "PENDING", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2763: trailing whitespace.
    "kind": "compute#operation", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2764: trailing whitespace.
    "name": "my_instance_template1", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2765: trailing whitespace.
    "insertTime": "2016-09-02T09:31:52.285-07:00", 
/home/erjohnso/ASF/libcloud/.git/rebase-apply/patch:2766: trailing whitespace.
    "targetLink": "https://www.googleapis.com/compute/v1/projects/project_name/global/instanceTemplates/my_instance_template1", 
warning: squelched 113 whitespace errors
warning: 118 lines add whitespace errors.

These errors are not super helpful in tracking them down, but I did confirm that there are trailing spaces in https://github.com/supertom/libcloud/blob/c332b347cc92bb1e67c02ff6e03d2620a09e68d4/libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json

Poking around a bit more, looks like it's been this way for a while in a few other spots.

$ find libcloud/test -name '*.json' -exec grep -l '.*\s$' {} \;
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_get.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_add.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_edit.json
libcloud/test/loadbalancer/fixtures/gogrid/loadbalancer_list.json
libcloud/test/loadbalancer/fixtures/gogrid/ip_list.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_3xxx.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_post.json
libcloud/test/loadbalancer/fixtures/rackspace/v1_slug_loadbalancers_8290_nodes.json
libcloud/test/dns/fixtures/auroradns/zone_example_com_records.json
libcloud/test/dns/fixtures/auroradns/zone_list.json
libcloud/test/container/fixtures/ecs/stoptask.json
libcloud/test/compute/fixtures/rimuhosting/r_orders.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps_running_state.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_new_vps.json
libcloud/test/compute/fixtures/rimuhosting/r_pricing_plans.json
libcloud/test/compute/fixtures/rimuhosting/r_orders_order_88833465_api_ivan_net_nz_vps.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_removeInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_global_instanceTemplates_insert.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_insert.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_addInstances.json
libcloud/test/compute/fixtures/gce/operations_operation_zones_us_central1_a_instanceGroups_myname_setNamedPorts.json
libcloud/test/compute/fixtures/gce/zones_us_central1_a_instanceGroups_myname_delete.json
libcloud/test/compute/fixtures/elastichosts/drives_info.json
libcloud/test/compute/fixtures/elastichosts/drives_create.json
libcloud/test/compute/fixtures/elastichosts/servers_create.json
libcloud/test/compute/fixtures/openstack_identity/v3/v3_users_a.json
libcloud/test/compute/fixtures/gogrid/lookup_list_ip_datacenter.json
libcloud/test/compute/fixtures/gogrid/password_list.json
libcloud/test/compute/fixtures/gogrid/image_save.json
libcloud/test/compute/fixtures/gogrid/ip_list.json
libcloud/test/compute/fixtures/ecp/vm_list.json
libcloud/test/compute/fixtures/ecp/vm_1_action_stop.json
libcloud/test/compute/fixtures/ecp/vm_1_get.json
libcloud/test/compute/fixtures/ecp/vm_1_action_start.json
libcloud/test/compute/fixtures/ecp/htemplate_list.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_groups_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_os_security_group_rules_create.json
libcloud/test/compute/fixtures/openstack_v1.1/_servers_1c01300f-ef97-4937-8f03-ac676d6234be_os-security-groups.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_a.json
libcloud/test/compute/fixtures/brightbox/list_images.json
libcloud/test/compute/fixtures/brightbox/create_cloud_ip.json
libcloud/test/compute/fixtures/brightbox/list_zones.json
libcloud/test/compute/fixtures/brightbox/list_server_types.json
libcloud/test/compute/fixtures/brightbox/list_servers.json
libcloud/test/compute/fixtures/brightbox/create_server_gb1_b.json
libcloud/test/compute/fixtures/brightbox/list_cloud_ips.json

I also upgraded packages on my dev machine, so it's possible this is new output (to me) that wasn't surfaced from my old dev packages.

@asfgit asfgit closed this in cffd964 Oct 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@supertom @erjohnso @tonybaloney