[LIBCLOUD-961] Update GCE public image projects list. #1143

smarlowucf · 2017-11-13T18:57:49Z

Update GCE public image projects list

Description

Add missing projects to list:

cos-cloud
suse-byos-cloud
suse-sap-cloud
windows-sql-cloud

Issue link: https://issues.apache.org/jira/browse/LIBCLOUD-961

Status

done, ready for review

Checklist (tick everything that applies)

Code linting (required, can be done after the PR checks)
Documentation
Tests
ICLA (required for bigger changes)

codecov-io · 2017-11-13T19:13:36Z

Codecov Report

Merging #1143 into trunk will increase coverage by 0.08%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##            trunk   #1143      +/-   ##
=========================================
+ Coverage   86.11%   86.2%   +0.08%     
=========================================
  Files         348     348              
  Lines       66891   67304     +413     
  Branches     5939    5958      +19     
=========================================
+ Hits        57602   58018     +416     
+ Misses       6859    6853       -6     
- Partials     2430    2433       +3

Impacted Files	Coverage Δ
libcloud/test/compute/test_gce.py	`97.73% <100%> (+0.08%)`	⬆️
libcloud/compute/drivers/gce.py	`76.64% <100%> (+0.37%)`	⬆️
libcloud/test/compute/test_ec2.py	`97.77% <0%> (-0.16%)`	⬇️
libcloud/compute/drivers/dimensiondata.py	`88.38% <0%> (-0.05%)`	⬇️
libcloud/compute/drivers/ec2.py	`74.25% <0%> (ø)`	⬆️
libcloud/compute/drivers/profitbricks.py	`69.58% <0%> (ø)`	⬆️
libcloud/test/common/test_openstack_identity.py	`97.94% <0%> (ø)`	⬆️
libcloud/test/loadbalancer/test_gogrid.py	`99.02% <0%> (ø)`	⬆️
libcloud/compute/types.py	`100% <0%> (ø)`	⬆️
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34d2f0a...7a267b8. Read the comment docs.

pquentin

Thank you for your contribution! I have two questions.

pquentin · 2018-01-31T05:11:30Z

libcloud/compute/drivers/gce.py

        "coreos-cloud": ["coreos"],
        "debian-cloud": ["debian", "backports"],
        "gce-nvme": ["nvme-backports"],
        "google-containers": ["container-vm"],
        "opensuse-cloud": ["opensuse"],
        "rhel-cloud": ["rhel"],
        "suse-cloud": ["sles", "suse"],
+        "suse-byos-cloud": ["sles", "suse"],


The list may have been updated since you submitted the pull request, but I don't see suse-byos-cloud in https://cloud.google.com/compute/docs/images#os-compute-support. Do you think it's still needed?

Yes, it is still needed. It's not in the list however in the SUSE details below it's mentioned:

If you need to reuse your existing SLES licenses to create Compute Engine instances, you can use Bring Your Own Subscription (BYOS) images available in the suse-byos-cloud project.

pquentin · 2018-01-31T05:13:11Z

libcloud/compute/drivers/gce.py

@@ -1733,15 +1733,19 @@ class GCENodeDriver(NodeDriver):

    IMAGE_PROJECTS = {
        "centos-cloud": ["centos"],
+        "cos-cloud": ["container-vm"],


Could we update the image families too? For example, in the list you linked, I see that the families associated with cos-cloud are cos-stable, cos-beta and cos-dev.

Yeah, I can update the image families.

On a side not: would it be possible to generate this list dynamically?

It can be interesting to generate the list dynamically if there's a JSON endpoint somewhere. I'm afraid crawling the HTML page is not worth the effort? I don't know.

Add missing projects and update the image families for each project based on `gcloud compute images list` info.

smarlowucf · 2018-02-16T18:02:32Z

@pquentin I've updated the list according gcloud tool. Removed the google containers project which is now cos-cloud and google nvme backports which doesn't appear in the project list anymore.

pquentin · 2018-02-25T16:21:10Z

Thanks! This looks good, but the tests need to be updated: see https://travis-ci.org/apache/libcloud/builds/342458915. Please tell me if you need help here.

coreos should point to coreos-alpha, coreos-beta, or coreos-stable.

smarlowucf · 2018-03-15T16:04:03Z

@pquentin I started updating tests but ran into mocking errors:

AttributeError: 'GCEMockHttp' object has no attribute '_global_images_family_coreos_stable'

Skimming docs I don't see anything regarding the mocking solution and how to go about updating/recording new json responses.

pquentin · 2018-03-15T16:48:55Z

@smarlowucf sorry this is something that we really need to document. See #1157 (comment) for an explanation in the context of another driver

pquentin · 2018-03-16T04:13:39Z

Does that make sense?

smarlowucf · 2018-03-16T18:53:22Z

Does that make sense?

@pquentin Yep, thanks. I will take a look.

smarlowucf · 2018-03-16T22:44:36Z

@pquentin The tests are passing but I'm not sure I did the fixtures correct. Also, this doesn't include any updated image lists for the new projects that were added.

Is there a way to record new json with image info for the new projects? gcloud doesn't seem to have a json option.

GwenaelPellenArkeup · 2018-03-20T16:07:55Z

@smarlowucf I may have misunderstood the question :

Is there a way to record new json with image info for the new projects? gcloud doesn't seem to have a json option.

gcloud can export result in json format.

$ gcloud compute images list --format=json --filter='(family: "coreos")'

[
  {
    "archiveSizeBytes": "500540800",
    "creationTimestamp": "2018-03-02T18:02:37.081-08:00",
    "description": "CoreOS, CoreOS alpha, 1702.1.0, amd64-usr published on 2018-03-03",
    "diskSizeGb": "9",
    "family": "coreos-alpha",
    "guestOsFeatures": [
      {
        "type": "VIRTIO_SCSI_MULTIQUEUE"
      }
    ],
    "id": "4918486452405779667",
    "kind": "compute#image",
    "labelFingerprint": "42WmSpB8rSM=",
    "licenses": [
      "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/licenses/coreos-alpha"
    ],
    "name": "coreos-alpha-1702-1-0-v20180303",
    "rawDisk": {
      "containerType": "TAR",
      "source": ""
    },
    "selfLink": "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/images/coreos-alpha-1702-1-0-v20180303",
    "sourceType": "RAW",
    "status": "READY"
  },
  {
    "archiveSizeBytes": "490827904",
    "creationTimestamp": "2018-03-09T10:34:21.565-08:00",
    "description": "CoreOS, CoreOS beta, 1688.3.0, amd64-usr published on 2018-03-09",
    "diskSizeGb": "9",
    "family": "coreos-beta",
    "guestOsFeatures": [
      {
        "type": "VIRTIO_SCSI_MULTIQUEUE"
      }
    ],
    "id": "6827356011088538947",
    "kind": "compute#image",
    "labelFingerprint": "42WmSpB8rSM=",
    "licenses": [
      "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/licenses/coreos-beta"
    ],
    "name": "coreos-beta-1688-3-0-v20180309",
    "rawDisk": {
      "containerType": "TAR",
      "source": ""
    },
    "selfLink": "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/images/coreos-beta-1688-3-0-v20180309",
    "sourceType": "RAW",
    "status": "READY"
  },
  {
    "archiveSizeBytes": "470601088",
    "creationTimestamp": "2018-02-15T09:50:52.929-08:00",
    "description": "CoreOS, CoreOS stable, 1632.3.0, amd64-usr published on 2018-02-15",
    "diskSizeGb": "9",
    "family": "coreos-stable",
    "guestOsFeatures": [
      {
        "type": "VIRTIO_SCSI_MULTIQUEUE"
      }
    ],
    "id": "6217468005063358099",
    "kind": "compute#image",
    "labelFingerprint": "42WmSpB8rSM=",
    "licenses": [
      "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/licenses/coreos-stable"
    ],
    "name": "coreos-stable-1632-3-0-v20180215",
    "rawDisk": {
      "containerType": "TAR",
      "source": ""
    },
    "selfLink": "https://www.googleapis.com/compute/v1/projects/coreos-cloud/global/images/coreos-stable-1632-3-0-v20180215",
    "sourceType": "RAW",
    "status": "READY"
  }
]

smarlowucf · 2018-03-20T23:18:50Z

gcloud can export result in json format.

@GwenaelPellenArkeup Ah, thanks. I must have overlooked the format option... Will give it another shot.

cos-cloud, suse-byos-cloud, suse-sap-cloud, windows-sql-cloud.

smarlowucf · 2018-03-21T15:15:23Z

@pquentin I've added json image lists for the new projects. If this all looks good I will go ahead and merge/cleanup commit history.

pquentin · 2018-03-21T17:22:00Z

Thanks! Merged in trunk. ✨ Really nice contribution.

I cleaned the history myself, I hope you don't mind. Can you please close LIBCLOUD-961?

smarlowucf · 2018-03-21T17:28:34Z

I cleaned the history myself, I hope you don't mind.

Yep, no worries. Thanks.

Can you please close LIBCLOUD-961?

Closed.

* 'trunk' of github.com:apache/libcloud: (1281 commits) Add changes for apache#1196 Added ability to define OS disk size for Azure ARM compute Add changes for apache#1198 Google Cloud DNS documentation test_s3: Fix lint Fix one GCE test fix spelling in OpenStack ex_files description Add changes for apache#1181 [LIBCLOUD-985] Add coverage for newer GCE network and subnet creation options. LIBCLOUD-986: Fix S3 iteration without prefix Add changes for apache#1195 guess s3 multipart upload content type. Add changes for apache#1193 LIBCLOUD-986: Add prefix option to azure blob list and iterate container methods Add changes for apache#1143 [LIBCLOUD-961] Update GCE public image list. Add changes for apache#1190 fix unexpected body type openstack image sharing Add changes for apache#1191 implement attach/detach floating IPs for DO ...

pquentin suggested changes Jan 31, 2018

View reviewed changes

[LIBCLOUD-961] Update GCE public image list.

a790af0

Add missing projects and update the image families for each project based on `gcloud compute images list` info.

Update compute unit tests image families.

25e56cb

coreos should point to coreos-alpha, coreos-beta, or coreos-stable.

pquentin mentioned this pull request Mar 16, 2018

Add "cos-cloud" project in IMAGE_PROJECTS #1189

Closed

4 tasks

Update fixtures and re-add opensuse-cloud.

b3490d1

Add json images list for new projects.

7a267b8

cos-cloud, suse-byos-cloud, suse-sap-cloud, windows-sql-cloud.

asfgit closed this in f8634a8 Mar 21, 2018

smarlowucf deleted the LIBCLOUD-961_image_projects branch August 14, 2018 19:20

XN137 mentioned this pull request Jul 4, 2019

GCE fails to find image for newer image families (i.e. ubuntu-1804-lts) #1303

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[LIBCLOUD-961] Update GCE public image projects list. #1143

[LIBCLOUD-961] Update GCE public image projects list. #1143

smarlowucf commented Nov 13, 2017 •

edited

codecov-io commented Nov 13, 2017 •

edited

pquentin left a comment

pquentin Jan 31, 2018

smarlowucf Feb 16, 2018

pquentin Jan 31, 2018

smarlowucf Feb 16, 2018

smarlowucf Feb 16, 2018

pquentin Feb 25, 2018

smarlowucf commented Feb 16, 2018

pquentin commented Feb 25, 2018

smarlowucf commented Mar 15, 2018

pquentin commented Mar 15, 2018

pquentin commented Mar 16, 2018

smarlowucf commented Mar 16, 2018

smarlowucf commented Mar 16, 2018

GwenaelPellenArkeup commented Mar 20, 2018

smarlowucf commented Mar 20, 2018 •

edited

smarlowucf commented Mar 21, 2018

pquentin commented Mar 21, 2018

smarlowucf commented Mar 21, 2018

[LIBCLOUD-961] Update GCE public image projects list. #1143

[LIBCLOUD-961] Update GCE public image projects list. #1143

Conversation

smarlowucf commented Nov 13, 2017 • edited

Update GCE public image projects list

Description

Status

Checklist (tick everything that applies)

codecov-io commented Nov 13, 2017 • edited

Codecov Report

pquentin left a comment

Choose a reason for hiding this comment

pquentin Jan 31, 2018

Choose a reason for hiding this comment

smarlowucf Feb 16, 2018

Choose a reason for hiding this comment

pquentin Jan 31, 2018

Choose a reason for hiding this comment

smarlowucf Feb 16, 2018

Choose a reason for hiding this comment

smarlowucf Feb 16, 2018

Choose a reason for hiding this comment

pquentin Feb 25, 2018

Choose a reason for hiding this comment

smarlowucf commented Feb 16, 2018

pquentin commented Feb 25, 2018

smarlowucf commented Mar 15, 2018

pquentin commented Mar 15, 2018

pquentin commented Mar 16, 2018

smarlowucf commented Mar 16, 2018

smarlowucf commented Mar 16, 2018

GwenaelPellenArkeup commented Mar 20, 2018

smarlowucf commented Mar 20, 2018 • edited

smarlowucf commented Mar 21, 2018

pquentin commented Mar 21, 2018

smarlowucf commented Mar 21, 2018

smarlowucf commented Nov 13, 2017 •

edited

codecov-io commented Nov 13, 2017 •

edited

smarlowucf commented Mar 20, 2018 •

edited