Systemvm template api #45

Spaceman1984 · 2019-09-02T12:32:26Z

Description

[WIP] System Template API
PR for review

Release notes: https://shapeblue.sharepoint.com/:w:/r/Shared%20Documents/Development/Roadmap/FR%20Documents/FR%2038%20-%20System%20VM%20Management%20API/FR38%20-%20Release%20Notes.docx?d=w5a674714278342f49ebe182e7aa0afd0&csf=1&e=Siq9bB

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):

How Has This Been Tested?

nvazquez · 2019-10-14T11:24:35Z

Ping for review @rhtyd, I'll be reviewing this PR. If you can review as well it will be great

nvazquez

Did a first review but could not finish it yet, leaving some comments ffrom my first review

nvazquez · 2019-10-15T03:10:58Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+
+        // mount locally
+        String mountPoint = "/tmp/nfsmount";
+        Script.runSimpleBashScript("mkdir " + mountPoint);


nvazquez · 2019-10-15T03:11:31Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+        for (String imageStore: imageStores){
+            try {
+                URI uri = new URI(imageStore);
+                Script.runSimpleBashScript("sudo mount -t nfs " + uri.getHost() + ":" + uri.getPath() + " " + mountPoint);


sudo not needed

In my environment, this script doesn't work without sudo

I checked other places in the code base, mount is called using sudo

nvazquez · 2019-10-15T03:12:11Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+            }
+
+            // now download the template to the image store for every hypervisor if not specified
+            if (this.hypervisor != null && this.url != null){


Use StringUtils.isNotBlank() for these kind of checks

I am specifically checking if these values were not passed in.

nvazquez · 2019-10-15T03:13:57Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+            // now download the template to the image store for every hypervisor if not specified
+            if (this.hypervisor != null && this.url != null){
+
+                if (this.hypervisor.equalsIgnoreCase("kvm") || this.hypervisor.equalsIgnoreCase("lxc")) {


Check Hypervisor.HypervisorType.getType(hypervisor) function. You can convert the String to a Hypervisor type and compare using that type

nvazquez · 2019-10-15T03:25:04Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+
+                this.templateId = _queryService.getSystemVMTemplateId(this);
+
+                HttpDirectTemplateDownloader httpDirectTemplateDownloader = new HttpDirectTemplateDownloader(url, templateId, mountPoint, null,null);


Do not use this class, been checking it is duplicated and unused. You can invoke wget URL -P mountPoint

nvazquez · 2019-10-15T03:27:21Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+
+    @Override
+    public void execute(){
+        ListResponse<ImageStoreResponse> imageStoreResponses = _queryService.searchForImageStores(new ListImageStoresCmd());


Maybe you can place all this logic on lines 56-71 to a new method on QueryService receiving a SeedSystemVMTemplateCmd parameter

nvazquez · 2019-10-15T03:27:38Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+        try {
+            URI uri = new URI(imageStoreResponse.getUrl());
+
+            Script.runSimpleBashScript("mkdir " + mountPoint);


-p and sudo below

nvazquez · 2019-10-15T03:28:49Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+
+        String script = Script.findScript("scripts/storage/secondary/","cloud-install-sys-tmplt");
+
+        String command = script + " -h " + hypervisor + " -F -m " + mountPoint + " -f " + uploadPath + "/" + fileUUID;


Maybe use StringFormat.format() for readability

nvazquez · 2019-10-15T03:30:03Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+            command += " -t " + templateId;
+        }
+
+        Script.runSimpleBashScriptForExitValue(command);


No checks on the exit value for the response?

nvazquez · 2019-10-15T03:32:47Z

engine/schema/src/main/java/com/cloud/upgrade/OfficialSystemVMTemplate.java

+
+import com.cloud.hypervisor.Hypervisor;
+
+public class OfficialSystemVMTemplate {


The whole class is using 4.11.3 instead of using a map between CloudStack version and the corresponding system VM template version

This code was added in an upgrade script, I moved it to a class.

Maybe out of scope but I think these maps can no longer be final if they are to be configurable by the admin and/or developer. (though of course the contents of the maps aren't)
Maybe provide an interface to do individual updates to it? (see other comment; the upgrade should now call the systemVM management API and not still do it its own way.
And finally, is this still needed in code (we now have administration in DB)?

rohityadavcloud · 2019-10-15T07:34:44Z

api/src/main/java/com/cloud/template/VirtualMachineTemplate.java

+        system, // system vm templates
+        inactive, // inactive system vm templates
        all // all templates (only usable by admins)


Can you explain why we're adding new template filters?

The "system" filter is used to get the system template id for seeding.
The "inactive" filter is used to list all inactive system vm templates. The list is used on the front end to list other system vm templates that are not currently used. We needed this view because the "all" filter doesn't show inactive system vm templates.

rohityadavcloud · 2019-10-15T07:35:41Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+        return id;
+    }
+
+    @Parameter(name = ApiConstants.ID, type = CommandType.UUID, entityType = ZoneResponse.class, required = true, description = "the ID of the zone")


Reformat the API implementation, see other/existing APIs how they're formatted.
We would for example prefer the params to be declared before the accessors (getters/setters); and then the execute() etc. methods.

rohityadavcloud · 2019-10-15T07:36:08Z

...in/java/org/apache/cloudstack/api/command/admin/storage/SeedOfficialSystemVMTemplateCmd.java

+import com.cloud.utils.http.downloader.HttpDirectTemplateDownloader;
+import com.cloud.utils.script.Script;
+
+@APICommand(name = "seedOfficialSystemVMTemplate", description = "Copies an official system vm template into secondary storage",


Could be simply rename this to 'registerSystemVMTemplate'?

rohityadavcloud · 2019-10-15T07:36:43Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+import com.cloud.utils.script.Script;
+
+
+@APICommand(name = "seedSystemVMTemplate", description = "Copies a system vm template into secondary storage",


Why do we have two separate APIs for the same thing?

rohityadavcloud · 2019-10-15T07:37:16Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+        hypervisor = hypervisor.toLowerCase();
+
+        String mountPoint = "/tmp/nfsmount";
+        String uploadPath = "/tmp/upload";


Move the logic to a manager class; treat APIs as containers of params; and execute() to simply process and send a response.

rohityadavcloud · 2019-10-15T07:37:42Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+
+    @Override
+    public String getCommandName() {
+        return "seedsystemvmtemplateresponse";


Refer to hackerbook and other APIs, on how to set the name. Use static API name.

rohityadavcloud

Left some comments.

borisstoyanov

@Spaceman1984 please address the comments in the integration tests section

borisstoyanov · 2019-10-15T13:37:13Z

test/integration/smoke/test_templates.py

 #Import System modules
 import time
-from marvin.cloudstackAPI import (createTemplate, listOsTypes)
+from marvin.cloudstackAPI import (createTemplate, listOsTypes, activateSystemVMTemplate)


why do we need changes in this file? I think the tests are in separate file?

Done, removed the unused import.

borisstoyanov · 2019-10-16T11:59:00Z

test/integration/smoke/test_activate_template.py

+
+    def test_02_copySystemVMtemplate(self):
+        # this test needs 2 zones, the template will be copied from 1 zone to the next
+        pass


This will report that test_02 is passing, if we cannot facilitate the execution of the test please remove it with propper comment, simply reporting that it's passing is wrong

Removed the test

borisstoyanov · 2019-10-16T12:05:11Z

test/integration/smoke/test_activate_template.py

@@ -0,0 +1,182 @@
+#!/usr/bin/env python


Perhaps it's worth adding tests for:
Register a template as systemvm template
Add tests for getSystemVMTemplateDefaultUrl - positive and negative
Investigate if upload template as system template is feasible/worth doing

Already registering a template in the "activate" test.
Added a test for getSystemVMTemplateDefaultUrl.
Uploading a template will need to be done using a form post. This is theoretically possible but would take some effort.

Please add at least a test for each new api and api change, we're now having 3 new apis and 3 old that have been changed

test/integration/smoke/test_activate_template.py

borisstoyanov · 2019-10-17T08:30:51Z

test/integration/smoke/test_activate_template.py

+            cls.apiclient,
+            cls.services["service_offerings"]["tiny"]
+        )
+        cls._cleanup.append(cls.service_offering)


Template stays after test run, cleanup does not seem to be cleaning all resources, can you verify all entities are deleted (accounts, domains, template

@Spaceman1984 note that the order of cleanup can matter at times.

cleanup_resources(cls.apiclient, reversed(cls._cleanup))

might help. Also make sure non-deletable items are cleaned in a separate method; think of settings or trust configurations and such. (not doing a full review, just stumbled upon this)

Removed added system VM template.

borisstoyanov · 2019-10-17T08:31:23Z

test/integration/smoke/test_activate_template.py

+            domainid=cls.domain.id
+        )
+        cls._cleanup.append(cls.user)
+        cls.service_offering = ServiceOffering.create(


Why do we need service offering? Do you plan to use it in VM deployment?

Leftover code from where I copied the test from, I have removed it.

test/integration/smoke/test_activate_template.py

borisstoyanov · 2019-10-25T06:53:42Z

test/integration/smoke/test_activate_template.py

+            for temp in cls.templates:
+                cmd = activateSystemVMTemplate.activateSystemVMTemplateCmd()
+                cmd.id = temp.id
+                cls.apiclient.activateSystemVMTemplate(cmd)


As far as I see there's no template being deleted here?

borisstoyanov · 2019-10-25T06:58:08Z

test/integration/smoke/test_activate_template.py

+        cls.unsupportedHypervisor = False
+        cls.hypervisor = cls.testClient.getHypervisorInfo()
+        if cls.hypervisor.lower() not in ['kvm']:
+            # Only testing kvm


why? This feature is agnostic to hypervisor, this test should be able to work with all hypervisors

borisstoyanov · 2019-10-25T07:01:13Z

test/integration/smoke/test_activate_template.py


        self.activateTemplateCmd = activateSystemVMTemplate.activateSystemVMTemplateCmd()
        self.listConfigurationsCmd = listConfigurations.listConfigurationsCmd()
        self.listConfigurationsCmd.name = 'router.template.kvm'


please do not hardcode, this feature is not just kvm specific

DaanHoogland

why the getUrl APIs?
Impressive bit of code, I reviewed until the test files (first 43 files)
looks mostly good.

DaanHoogland · 2019-10-29T13:33:56Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+        authorized = {RoleType.Admin})
+public class SeedSystemVMTemplateCmd extends BaseCmd {
+
+    private static final String s_name = "seedsystemvmtemplateresponse";


better use a name without response, use it in the annotation parameter APICommand.name and construct the commandsname with s_name (NAME is a better name) .toLower() + "response".
there are plenty examples about. (also of this older way of doing things admittedly)

DaanHoogland · 2019-10-29T13:34:37Z

api/src/main/java/org/apache/cloudstack/api/command/admin/storage/SeedSystemVMTemplateCmd.java

+import com.cloud.utils.script.Script;
+
+
+@APICommand(name = "seedSystemVMTemplate", description = "Copies a system vm template into secondary storage",


DaanHoogland · 2019-10-29T13:40:46Z

api/src/main/java/org/apache/cloudstack/api/command/user/template/RegisterTemplateCmd.java

                    "Both zoneid and zoneids cannot be specified at the same time");

-        if (zoneId == null && (zoneIds == null || zoneIds.isEmpty()))
+        if ((zoneId == null && (zoneIds == null || zoneIds.isEmpty())) && !isSystem())


why this exemption? I don't see why systemVM templates can be region wide and user VM templates couldn't

DaanHoogland · 2019-10-29T13:45:02Z

.../main/java/org/apache/cloudstack/api/command/admin/template/ActivateSystemVMTemplateCmd.java

+@APICommand(name = ActivateSystemVMTemplateCmd.APINAME,
+        description = "Activates an existing system virtual machine template to be used by CloudStack to create system virtual machines.",
+        responseObject = ActivateSystemVMTemplateResponse.class,
+        authorized = {RoleType.Admin})
+public class ActivateSystemVMTemplateCmd extends BaseCmd {


is there (a plan for) a deactive. Deleting templates is not trivial as these may be used still but still we want to make sure we can in the long run.

DaanHoogland · 2019-10-29T13:46:48Z

engine/schema/src/main/java/com/cloud/storage/VMTemplateVO.java

+        if (type == TemplateType.SYSTEM){
+            uniqueName = "routing-" + id;
+        }


not all systemVMs are routing. is this a sledge hammer to a fine nail? might be needed but it deserves a comment for future generations.

DaanHoogland · 2019-10-29T14:33:36Z

server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java

+                if (!_statsCollector.imageStoreHasEnoughCapacity(imageStore) &&
+                        (template.getTemplateType().equals(Storage.TemplateType.SYSTEM) && !capacityChecker.hasEnoughCapacity(imageStore))) {


Are we checking for capacity in two ways?

DaanHoogland · 2019-10-29T14:36:50Z

server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java

+                                    " is disabled, so uploading template to management server" + imageStore.getId());
+                            //continue;


don't like this change:

space removed at end of string so imagestore id gets glued to the word server.

string says management server biut then it give not the id of the server but on the store, without explanation.

// continue ==> code in comment, please remove or uncomment

DaanHoogland · 2019-10-29T14:37:40Z

server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java

+
+

superfluent empty lines

DaanHoogland · 2019-10-29T14:38:45Z

server/src/main/java/com/cloud/template/HypervisorTemplateAdapter.java

        List<Long> zoneIdList = profile.getZoneIdList();
-
-        if (template.getTemplateType() == TemplateType.SYSTEM) {
+        // TODO: Add a check to see if this template is used by console proxy, router or storagevm


these are TemplateType.SYSTEM as well aren't they?

DaanHoogland · 2019-10-29T14:43:37Z

server/src/main/java/com/cloud/template/TemplateAdapterBase.java

        List<VMTemplateVO> systemvmTmplts = _tmpltDao.listAllSystemVMTemplates();
        for (VMTemplateVO template : systemvmTmplts) {
-            if (template.getName().equalsIgnoreCase(name) || template.getDisplayText().equalsIgnoreCase(displayText)) {
+            if (template.getTemplateType() != TemplateType.SYSTEM && (template.getName().equalsIgnoreCase(name) || template.getDisplayText().equalsIgnoreCase(displayText))) {


30 and 10 lines into a 130 lines method some things have changed! On cannot but love maintaining monolithic code with hundreds of people.
No idea what else to say about these two changes ;)

borisstoyanov · 2019-10-31T10:13:43Z

test/integration/smoke/test_activate_template.py

+            if cls.firstRun:
+                # Save currently active template
+                activeTemplate = cls.apiclient.listTemplates(cls.listTemplatesCmd)[0]
+                cmd = deleteTemplate.deleteTemplateCmd()


deleteTemaplate?

Yes, we need to delete the newly created template as part of the cleanup.

borisstoyanov · 2019-10-31T12:14:30Z

test/integration/smoke/test_activate_template.py

+        cls._cleanup.append(cls.user)
+
+        cls.listTemplatesCmd = listTemplates.listTemplatesCmd()
+        cls.listTemplatesCmd.templatefilter= 'system'


isn't this going to return just a bunch of templates of type 'system'. Shouldn't we be looking to get the value from the global settings?

I'll see what I can do

This call only returns the currently active system VM template

borisstoyanov · 2019-10-31T12:51:02Z

test/integration/smoke/test_activate_template.py

+        self.test_template.name = 'test-system-' + self.hypervisor + '-4.11.3'
+        self.test_template.displaytext = 'test-system-' + self.hypervisor + '-4.11.3'
+        self.test_template.url = urlResponse.url.url #"http://download.cloudstack.org/systemvm/4.11/systemvmtemplate-4.11.3-kvm.qcow2.bz2"
+        self.test_template.format = "QCOW2"


format should be hypervisor specific also

I think it would be good to return the template file type as part of the getsystemVMTemplatedefaultURL call. I will add this.

borisstoyanov · 2019-11-07T13:40:25Z

test/integration/smoke/test_activate_template.py

+
+    @attr(tags=["advanced", "smoke"], required_hardware="true")
+    def test_03_seedSystemVMTemplate(self):
+        activeTemplate = self.apiclient.listTemplates(self.listTemplatesCmd)[0]


no need of this variable, cmd.templateid later can call directly the listTemplates api, and what if we have multiple active templates, how do we know which one we're dealing with? Probably include the given hypervisor.

* TODO: Automated tests development and UI integration

…emvm template will be whatever is stored in the database.

…es not being removed when adding system templates from the wizzard.

…flag to zone multi select box

rohityadavcloud · 2021-12-23T08:06:38Z

Abandoned.

* Prevent addition of duplicate PF rules on scale up and no rules left behind on scale down (#32) * fix missing dependency injection * NSX: Fix concurrency issues on port forwarding rules deletion (#37) * Fix concurrency issues on port forwarding rules deletion * Refactor objectExists * Fix unit test * Fix test * Small fixes * CKS: Externalize control and worker node setup wait time and installation attempts (#38) * NSX: Add shared network support (#41) * NSX: Fix number of physical networks for Guest traffic checks and leftover rules on CKS cluster deletion (#45) * Fix pf rules removal on CKS cluster deletion * Fix check for number of physical networks for guest traffic * Fix unit test * fix logger * NSX: Handle CheckHealthCommand to avoid host disconnection and errors on APIs * NSX: Handle CheckHealthCommand to avoid host disconnection and errors on APIs * Remove unused string * fix logger * Update UDP active monitor to ICMP * Fix NPE on restarting VPC with additional public IPs * NSX / VPC: Reuse Source NAT IP from systemVM range on restarts * CKS: Public IP not found for VPC networks * Externalize retries and inverval for NSX segment deletion (#67) * remove unused import * remove duplicate imports * remove unused import * revert externalizing cks settings * fix test * Refactor log messages * Address comments * Fix issue caused due to forward merge: 90fe1d --------- Co-authored-by: Nicolas Vazquez <nicovazquez90@gmail.com> Co-authored-by: Rohit Yadav <rohit.yadav@shapeblue.com>

nvazquez reviewed Oct 15, 2019

View reviewed changes

rohityadavcloud reviewed Oct 15, 2019

View reviewed changes

rohityadavcloud requested changes Oct 15, 2019

View reviewed changes

borisstoyanov requested changes Oct 16, 2019

View reviewed changes

borisstoyanov reviewed Oct 17, 2019

View reviewed changes

test/integration/smoke/test_activate_template.py Outdated Show resolved Hide resolved

borisstoyanov reviewed Oct 17, 2019

View reviewed changes

test/integration/smoke/test_activate_template.py Show resolved Hide resolved

borisstoyanov reviewed Oct 17, 2019

View reviewed changes

test/integration/smoke/test_activate_template.py Show resolved Hide resolved

borisstoyanov reviewed Oct 25, 2019

View reviewed changes

DaanHoogland reviewed Oct 29, 2019

View reviewed changes

borisstoyanov reviewed Oct 31, 2019

View reviewed changes

borisstoyanov reviewed Nov 7, 2019

View reviewed changes

Spaceman1984 force-pushed the master branch from 6bde2f6 to 29e1bbc Compare December 20, 2019 09:25

Spaceman1984 force-pushed the systemvm-template-api branch 2 times, most recently from a1b2bf8 to dd2dd59 Compare January 17, 2020 06:53

ernjvr and others added 4 commits January 31, 2020 15:45

* Compete back-end implementation pending full feature test coverage

584ae3f

* TODO: Automated tests development and UI integration

* Compete back-end implementation pending full feature test coverage

b9630bf

* TODO: Automated tests development and UI integration

Organised imports

6d58996

Work in progress

1936846

Spaceman1984 added 22 commits January 31, 2020 15:52

Review changes

62cd45a

Removed whitespace

1b6e552

Rearranged cleanup in tests

0ed43f4

Fixed template not activating after upload

db5cff5

Changed temp folder location for template download

559a38a

Getting around file creation issues for template.properties

c706e3b

Fixed incorrect hypervisor selected for official template

d02e392

Fixed upload from local on zone wizard

4a29de4

Removed getDefault url test. This test is not needed because the syst…

eaf0303

…emvm template will be whatever is stored in the database.

Fixed imports and whitespace

9133d04

Removed file after rebase

6c69a96

Raised systemvmteplate copy timeout to 15 minutes

c6b95c1

Raised timeouts in seedtemplate command

d22f341

Fixed null pointer on uploading iso from local and old system templat…

051f5e4

…es not being removed when adding system templates from the wizzard.

Fixed activate template behaviour, improved error handling on wizzard

7af02f1

Added deprecated script warning

1773688

Fixed duplicate bean definition

1066601

Fixed file upload error when using advanced zone

b423f33

Fixed activate template visible for user template and added required …

909d04f

…flag to zone multi select box

Renamed id to zoneid on seed template api

cea0ec2

fixed failing delete_template test

5df90fe

Post rebase fix

d8532b9

Spaceman1984 force-pushed the systemvm-template-api branch from 02b9c24 to d8532b9 Compare February 3, 2020 08:05

Merge remote-tracking branch 'origin/master' into systemvm-template-api

c8c1130

rohityadavcloud closed this in cba4d6d Jan 20, 2021

rohityadavcloud reopened this Jan 20, 2021

rohityadavcloud closed this Dec 23, 2021


		this.templateId = _queryService.getSystemVMTemplateId(this);

		HttpDirectTemplateDownloader httpDirectTemplateDownloader = new HttpDirectTemplateDownloader(url, templateId, mountPoint, null,null);


		String script = Script.findScript("scripts/storage/secondary/","cloud-install-sys-tmplt");

		String command = script + " -h " + hypervisor + " -F -m " + mountPoint + " -f " + uploadPath + "/" + fileUUID;


		import com.cloud.hypervisor.Hypervisor;

		public class OfficialSystemVMTemplate {

		import com.cloud.utils.script.Script;


		@APICommand(name = "seedSystemVMTemplate", description = "Copies a system vm template into secondary storage",

		if (!_statsCollector.imageStoreHasEnoughCapacity(imageStore) &&
		(template.getTemplateType().equals(Storage.TemplateType.SYSTEM) && !capacityChecker.hasEnoughCapacity(imageStore))) {

		" is disabled, so uploading template to management server" + imageStore.getId());
		//continue;

Systemvm template api #45

Systemvm template api #45

Uh oh!

Conversation

Spaceman1984 commented Sep 2, 2019 • edited by nvazquez Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

nvazquez commented Oct 14, 2019

Uh oh!

nvazquez left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Spaceman1984 Oct 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohityadavcloud left a comment

Choose a reason for hiding this comment

Uh oh!

borisstoyanov left a comment

Choose a reason for hiding this comment

Uh oh!

borisstoyanov Oct 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Spaceman1984 commented Sep 2, 2019 •

edited by nvazquez

Loading

Spaceman1984 Oct 28, 2019 •

edited

Loading

borisstoyanov Oct 15, 2019 •

edited

Loading