Gce image support #7080

bentsi · 2020-08-20T21:41:54Z

Add necessary changes to scylla_util.py in order to support Scylla Machine Image in GCE.
Fixes and improvements for curl function.

avikivity · 2020-08-22T18:48:00Z

dist/common/scripts/scylla_util.py

 # @param headers dict of k:v
-def curl(url, byte=False, headers={}):
+def curl(url, headers, byte=False):


I don't understand this change. Why switch the order around? Why make headers mandatory?

usually, the order of arguments is chosen by their use, importance, and common use case
example:

curl(url)

curl(url, headers)
and if we want to encode the output as byte string we set byte=True

curl(url, byte=True)

curl(url, headers, byte=True)
this is a better API.

Regarding making headers mandatory you are right, I will change it (but we can't use a mutable {} as a default (https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments)

And what about the removal of the default?

The goal is not to confuse the maintainer with lots of unrelated changes.

I am fixing a bug, we can't use a mutable ({}) as a default argument to the function (https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments)
I can separate it as a different commit.

avikivity · 2020-08-22T18:48:13Z

dist/common/scripts/scylla_util.py

            time.sleep(5)
            retries += 1
-            if (retries >= max_retries):
+            if retries >= max_retries:
                raise


These changes look unrelated.

@bentsi You can add another patch for this kind of style change.

this is styling fix, no need to put the round brackets here...

moved to separate commit

avikivity · 2020-08-25T14:12:22Z

dist/common/scripts/scylla_util.py

 # @param headers dict of k:v
-def curl(url, byte=False, headers={}):
+def curl(url, headers=None, byte=False):


Why not write headers={}?

The subject of the commit is wrong, since we could pass headers before. As fat as I can tell, this commit does nothing.

I will fix the commit message. It was originally done before Lubus changes that added the headers argument.

Thanks. This way I understand the change, and it can be backported independently if needed.

Avi, FYI doing headers={} in a function signature, can bite someone very hard,
since this dict is created only once. it basically becomes a global variable.

Yes, unexpected for a nice language such as Python.

avikivity · 2020-08-25T14:13:13Z

dist/common/scripts/scylla_util.py

-        return curl("http://169.254.169.254/computeMetadata/v1/instance/" + path+"?recursive=true", headers={"Metadata-Flavor": "Google"})
-        #169.254.169.254 is metadata.google.internal
+    def __instance_metadata(self, path, recursive=False):
+        return curl(self.META_DATA_BASE_URL + path + "?recursive=%s" % str(recursive).lower(),


Why did you remove the comment?

after the change, the information in the comment is not useful here

avikivity · 2020-08-25T14:14:25Z

dist/common/scripts/scylla_util.py

+def is_ec2():
+    patterns_files = ((r"^Amazon EC2$", "/sys/class/dmi/id/board_vendor"),
+                      (r"^ec2.*", "/sys/hypervisor/uuid"))
+    return match_patterns_in_files(patterns_files)


This looks fragile. Is there no way to do this from the metadata server?

this is what AWS recommends. You can read more here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html

The second one is not accurate. There is a 1:4000 chance of a random uuid starting with "ec2". They even document it as "probably".

I think it's better to use the instance metadata, since we have to read it anyway.

avikivity · 2020-08-26T10:22:26Z

dist/common/scripts/scylla_util.py

    max_retries = 5
    retries = 0
    while True:
        try:
-            req = urllib.request.Request(url,headers=headers)
+            req = urllib.request.Request(url, headers={} if not headers else headers)


You could write this as headers or {}

right. This will look better. Fixed.

slivne · 2020-09-14T10:04:09Z

@avikivity / @syuu1228 - according to bentsi he is fixed everything - please review and approve so we can move forward

fruch · 2020-09-14T10:14:11Z

dist/common/scripts/scylla_util.py

@@ -369,6 +394,14 @@ def __init__(self):
        self._type = self.__instance_metadata("instance-type")
        self.__populate_disks()

+    def is_aws_instance(self):


why is_gce_instance is static, and this one isn't ?

or at least a class method

the whole scylla_util lacks basic architecture and basic usage of OOP design principles (Base class in this case). I am not going to redesign and rewrite this...
I can't use static method since I need META_DATA_BASE_URL

I can use classmethod

syuu1228 · 2020-09-14T10:17:03Z

dist/common/scripts/scylla_util.py

-    elif os.path.exists('/sys/class/dmi/id/board_vendor'):
-        with open('/sys/class/dmi/id/board_vendor') as f:
-            s = f.read()
-        return True if re.match(r'^Amazon EC2$', s) else False


Why do we need to remove is_ec2() to support GCE?
There are multiple scripts relies on the function, if you remove this you will need to fix them too.

right, my bad, deleted it mistakenly when resolving conflicts with Lubush changes

slivne · 2020-09-15T11:17:17Z

@bentsi ping

…ction - It was set to {} that is incorrect and can lead to unexpected behavior https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments - Order of the arguments changed to more convinient way

- rename deprecated logging.warn to logging.warning - remove redundant round brackets in the if statement

bentsi · 2020-09-16T08:35:59Z

@slivne all comments resolved

avikivity · 2020-09-16T08:38:23Z

@syuu1228 @fruch please re-review.

fruch

Looks o.k.

tarzanek · 2020-11-18T10:37:38Z

dist/common/scripts/scylla_util.py

    return False


+def is_ec2():
+    return aws_instance.is_aws_instance()


so above code depended on /sys
this new code depends on http call to a service ...

I actually like the check for /sys much better, since no network is involved
shouldn't we keep the /sys check and use it before the http metadata call?
(I didn't do it for GCP machines, since I didn't find a way to do it, but since we have it for AWS, why making this check slower by adding network to the game?)

also GCP actually just does dns query (but I'd love it to be optimized and use /sys or any other variable instead)

@tarzanek That's how it was originally implemented but Avi was not happy about it:
comment from above:

so how will this method then behave on non cloud servers? or it will just wait for timeout?

in the current implementation, it will wait for a timeout... but it's not critical since those functions are mainly used in the Scylla Amazon/GCE images

avikivity reviewed Aug 22, 2020

View reviewed changes

bentsi force-pushed the gce-image-support branch 3 times, most recently from 354fae6 to bba96ef Compare August 24, 2020 21:47

avikivity reviewed Aug 25, 2020

View reviewed changes

bentsi force-pushed the gce-image-support branch from bba96ef to d08f077 Compare August 26, 2020 09:58

avikivity reviewed Aug 26, 2020

View reviewed changes

bentsi force-pushed the gce-image-support branch 3 times, most recently from f2e8df9 to 16f8796 Compare August 27, 2020 23:10

slivne requested review from avikivity and amoskong September 14, 2020 10:03

fruch reviewed Sep 14, 2020

View reviewed changes

fruch mentioned this pull request Sep 14, 2020

feature: GCE image support scylladb/scylla-machine-image#52

Merged

syuu1228 reviewed Sep 14, 2020

View reviewed changes

Bentsi Magidovich added 5 commits September 16, 2020 11:25

scylla_util.py: change default value for headers argument in curl fun…

a24ec26

…ction - It was set to {} that is incorrect and can lead to unexpected behavior https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments - Order of the arguments changed to more convinient way

scylla_util.py: styling fixes to curl function

f5d97af

- rename deprecated logging.warn to logging.warning - remove redundant round brackets in the if statement

scylla_util.py: adding timeout to curl function

1f31e97

scylla_util.py: make max_retries as a curl function argument

5aef44f

scylla_util.py: added GCE instance/image support

12e018a

bentsi force-pushed the gce-image-support branch from 16f8796 to 12e018a Compare September 16, 2020 08:33

slivne requested review from syuu1228 and fruch September 16, 2020 15:10

fruch approved these changes Sep 16, 2020

View reviewed changes

scylladb-promoter closed this in 809a13d Sep 17, 2020

scylladb-promoter merged commit 809a13d into scylladb:master Sep 17, 2020

bentsi mentioned this pull request Oct 11, 2020

fix: incorrect AWS user-data url #7388

Closed

tarzanek mentioned this pull request Nov 18, 2020

scylla_util.py: increase disk to ram ratio for GCP #7600

Closed

tarzanek reviewed Nov 18, 2020

View reviewed changes

Gce image support #7080

Gce image support #7080

Conversation

bentsi commented Aug 20, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slivne commented Sep 14, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slivne commented Sep 15, 2020

bentsi commented Sep 16, 2020

avikivity commented Sep 16, 2020

fruch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bentsi commented Aug 20, 2020 •

edited

Loading