Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CLOUDSTACK-10197: Rename xentools iso for XenServer 7.0+ #2365

Merged
merged 1 commit into from Jan 7, 2018

Conversation

khos2ow
Copy link
Contributor

@khos2ow khos2ow commented Dec 18, 2017

The xentools iso has been renamed from xs-tools to guest-tools
starting from XenServer 7.0.

This PR is the follow up of #2346

@reviewers: The main change is in getIsoVDIByURL method [line ~2589] the rest is autoformatting.

@rohityadavcloud
Copy link
Member

Lgtm
@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1408

@rohityadavcloud
Copy link
Member

@blueorangutan test centos7 xenserver-65sp1

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + xenserver-65sp1) has been kicked to run smoke tests

@rohityadavcloud rohityadavcloud added this to the 4.11 milestone Dec 19, 2017
@blueorangutan
Copy link

Trillian test result (tid-1822)
Environment: xenserver-65sp1 (x2), Advanced Networking with Mgmt server 7
Total time taken: 40212 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2365-t1822-xenserver-65sp1.zip
Smoke tests completed. 61 look OK, 6 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_04_extract_Iso Failure 5.21 test_iso.py
test_01_vpc_privategw_acl Failure 91.89 test_privategw_acl.py
test_02_vpc_privategw_static_routes Failure 298.83 test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup Failure 480.51 test_privategw_acl.py
test_04_rvpc_privategw_static_routes Failure 646.04 test_privategw_acl.py
test_02_create_template_with_checksum_sha1 Error 5.22 test_templates.py
test_03_create_template_with_checksum_sha256 Error 5.21 test_templates.py
test_04_create_template_with_checksum_md5 Error 5.21 test_templates.py
test_04_extract_template Failure 5.16 test_templates.py
ContextSuite context=TestISOUsage>:setup Error 0.00 test_usage.py
test_01_volume_usage Error 15.75 test_usage.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Failure 465.68 test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics Failure 407.38 test_vpc_redundant.py
test_05_rvpc_multi_tiers Failure 412.43 test_vpc_redundant.py
test_01_vpc_remote_access_vpn Error 166.47 test_vpc_vpn.py

@rohityadavcloud
Copy link
Member

I see new iso related failures, rest are known issues. Can you check @khos2ow

@khos2ow
Copy link
Contributor Author

khos2ow commented Dec 19, 2017

@rhtyd is there a way to see the error log?

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Dec 19, 2017

@khos2ow yes, download the marvin logs zip file from the test above, extract and see the logs for that test to get hints etc.

@khos2ow
Copy link
Contributor Author

khos2ow commented Dec 19, 2017

@rhtyd damn! I haven't seen that link!!! thanks.

@rohityadavcloud
Copy link
Member

@khos2ow can you fix the build failures?

@khos2ow khos2ow force-pushed the xentools-rename branch 2 times, most recently from 262b24a to 249378c Compare December 20, 2017 18:27
@rohityadavcloud
Copy link
Member

@khos2ow let me if you've addressed the issues, so I can kick another round of testing?

@khos2ow
Copy link
Contributor Author

khos2ow commented Dec 20, 2017

@rhtyd should be fine now, there was a code-style issue which I fixed it.

@rohityadavcloud
Copy link
Member

Okay
@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1449

@rohityadavcloud
Copy link
Member

@blueorangutan test centos7 xenserver-65sp1

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + xenserver-65sp1) has been kicked to run smoke tests

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Dec 21, 2017

@khos2ow the new changes causes management server to fail on start, in a fresh installation. I get:

2017-12-21 12:47:53,542 INFO  [c.c.h.x.r.XenServerConnectionPool] (main:null) (logid:) XenServer Connection Pool Configs: sleep.interval.on.error=10000
2017-12-21 12:47:53,568 DEBUG [c.c.a.m.AgentManagerImpl] (main:null) (logid:) Registering listener XcpServerDiscoverer with id 25
2017-12-21 12:47:53,590 DEBUG [c.c.u.d.T.Transaction] (main:null) (logid:) Rolling back the transaction: Time = 4 Name =  persist; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy109.persist:-1-XcpServerDiscoverer.createXsToolsISO:550-XcpServerDiscoverer.configure:500-CloudStackExtendedLifeCycle$3.with:115
2017-12-21 12:47:53,591 WARN  [o.a.c.s.m.c.ResourceApplicationContext] (main:null) (logid:) Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@1f60c30f: INSERT INTO vm_template (vm_template.id, vm_template.format, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.type, vm_template.url, vm_template.hvm, vm_template.bits, vm_template.created, vm_template.account_id, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.guest_os_id, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zones, vm_template.hypervisor_type, vm_template.extractable, vm_template.source_template_id, vm_template.state, vm_template.template_tag, vm_template.uuid, vm_template.sort_key, vm_template.enable_sshkey, vm_template.size, vm_template.update_count, vm_template.updated, vm_template.dynamically_scalable) VALUES (201, 'ISO', null, null, 1, 1, 'PERHOST', null, 1, 64, '2017-12-21 12:47:53', 1, null, _binary'xen-pv-drv-iso', 0, 1, 0, 0, 0, 'XenServer', 1, null, 'Active', null, _binary'a17c3900-50a7-4847-ab10-bdf309245aaf', 0, 0, null, 0, null, 0)
2017-12-21 12:47:53,592 WARN  [o.e.j.w.WebAppContext] (main:null) (logid:) Failed startup of context o.e.j.w.WebAppContext@49c43f4e{/client,file:///usr/share/cloudstack-management/webapp/,UNAVAILABLE}{/usr/share/cloudstack-management/webapp}
org.springframework.context.ApplicationContextException: Failed to start bean 'cloudStackLifeCycle'; nested exception is com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.jdbc.JDBC4PreparedStatement@1f60c30f: INSERT INTO vm_template (vm_template.id, vm_template.format, vm_template.unique_name, vm_template.name, vm_template.public, vm_template.featured, vm_template.type, vm_template.url, vm_template.hvm, vm_template.bits, vm_template.created, vm_template.account_id, vm_template.checksum, vm_template.display_text, vm_template.enable_password, vm_template.guest_os_id, vm_template.bootable, vm_template.prepopulate, vm_template.cross_zones, vm_template.hypervisor_type, vm_template.extractable, vm_template.source_template_id, vm_template.state, vm_template.template_tag, vm_template.uuid, vm_template.sort_key, vm_template.enable_sshkey, vm_template.size, vm_template.update_count, vm_template.updated, vm_template.dynamically_scalable) VALUES (201, 'ISO', null, null, 1, 1, 'PERHOST', null, 1, 64, '2017-12-21 12:47:53', 1, null, _binary'xen-pv-drv-iso', 0, 1, 0, 0, 0, 'XenServer', 1, null, 'Active', null, _binary'a17c3900-50a7-4847-ab10-bdf309245aaf', 0, 0, null, 0, null, 0)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:186)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:52)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:358)
        at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:159)
        at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123)

Please fix and test in your environment.

Copy link
Member

@rohityadavcloud rohityadavcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new changes break management server install/start, please fix.

@@ -529,13 +540,12 @@ public boolean processCommands(long agentId, long seq, Command[] commands) {
}

private void createXsToolsISO() {
String isoName = "xs-tools.iso";
VMTemplateVO tmplt = _tmpltDao.findByTemplateName(isoName);
VMTemplateVO tmplt = _tmpltDao.findByTemplateName(_toolsIsoName);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is at this line _toolsIsoName is not initialized yet (basically cretaeXsToolsISO is executed before newly added determineToolsIsoName method). My other idea is at this line we can register both xs-tools.iso and guest-tools.iso and either:

  • later on (for example in createServerResource method) delete/deactivate the wrong iso
  • present both of them on UI with different names and let the user select the correct iso based on the version of their Xen

@rhtyd @wido @syed @rafaelweingartner @swill any comment, suggestion, concern?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would start with a different question. Do we need to check which XenServers versions are used? I mean, Can’t we simply persist both ISOs in the database? Without needing to use that “determine” method.

It seems that as long as we have both entries in the database (or new ones in the far far future ahead of us), your logic in CitrixResourceBase.java would take care of the rest. I would recommend one thing though. You have the strings "guest-tools" and "xs-tools" in two different classes (at least the ones I am seeing in this PR). What about extracting them to an enum?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we do, that's the whole point of this PR. The tools ISO has been renamed. So the question will be we auto detect it or provide the options for user to select.

+1 on extracting to enum though, will do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we create a more transparent mode? I mean, the user sees just the "Xen-server-tools", and then underneath we select the right name according to the XenServer version.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's what I was trying to do but the way I did it was wrong, hence the first option mentioned above:

we can register both xs-tools.iso and guest-tools.iso and ... later on (for example in createServerResource method) delete/deactivate the wrong iso

Long id;
if (tmplt == null) {
id = _tmpltDao.getNextInSequence(Long.class, "id");
VMTemplateVO template =
VMTemplateVO.createPreHostIso(id, isoName, isoName, ImageFormat.ISO, true, true, TemplateType.PERHOST, null, null, true, 64, Account.ACCOUNT_ID_SYSTEM,
VMTemplateVO.createPreHostIso(id, _toolsIsoName, _toolsIsoName, ImageFormat.ISO, true, true, TemplateType.PERHOST, null, null, true, 64, Account.ACCOUNT_ID_SYSTEM,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not know what you think about the use of "_" here to denote object attributes, but I do not see any benefit on using it. As a matter of fact, I see most PRs removing them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I, honestly, don't like it and have never used it like this, but wanted to follow the same pattern in the code base.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is always good to create a new trend :)

if (isoURL.startsWith("xs-tools")) {
// XenServer 7.0+ => guest-tools.iso
// XenServer [other] => xs-tools.iso
if (isoURL.startsWith("xs-tools") || isoURL.startsWith("guest-tools")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible for isoURL to be null? If not, it is ok the way it is. Otherwise, it might be a good idea to use StringUtils.startsWith

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 I doubt it will be null, but checking for null almost always is good. I'll add it.

@rohityadavcloud
Copy link
Member

@khos2ow do make suitable changes and let me know when you next want to kick tests.

@khos2ow
Copy link
Contributor Author

khos2ow commented Jan 3, 2018

@rhtyd @rafaelweingartner the new change:

  • registers both iso names at ComponentLifecycle#configure time
  • and later on createServerResource removes the wrong one

Any comments, suggestions?

@rafaelweingartner
Copy link
Member

What are the assumptions here?
I mean, is it possible to have clusters/hosts with different versions of XenServer at the same time?
If so, we should work with both naming conventions at the same time, right?

What about something different? The user does not need to know the version of XenServer the cloud providers use. We could present a “dummy” ISO option called “XenServer tools. Then, underneath, when plugging the ISO to the VM, we check the host the VM is running and select/find/discover the right VDI that represent the XenServer tools ISO to be connected to the VM (according to host’s XenServer version). This would require only a single entry for any type of XenServer tools.

@khos2ow
Copy link
Contributor Author

khos2ow commented Jan 3, 2018

@rafaelweingartner I don't know if we can have a clusters/hosts with different XS versions, but in any case I changed it again as follow

  • register only one iso with name of xs-tools.iso and display_text of XenServer Tools Installer ISO (xen-pv-drv-iso)
  • get the actual iso name (either xs-tools.iso or guest-tools.is) on the fly based on product_version of com.xensource.xenapi.Host.Record

in the future, if the name of iso changes again, only another else block will be needed in CitrixResourceBase#actualIsoTemplate

The xentools iso has been renamed from xs-tools to guest-tools
starting from XenServer 7.0.
Copy link
Member

@rohityadavcloud rohityadavcloud left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM subject to testing

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1561

@rohityadavcloud
Copy link
Member

@blueorangutan test centos7 xenserver-65sp1

@blueorangutan
Copy link

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + xenserver-65sp1) has been kicked to run smoke tests

Copy link
Member

@rafaelweingartner rafaelweingartner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@khos2ow I liked this solution better.
I pointed out some remarks, could you take a look at them?

@@ -2592,9 +2592,10 @@ public VDI getIsoVDIByURL(final Connection conn, final String vmName, final Stri
String mountpoint = null;
if (isoURL.startsWith("xs-tools")) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This ISO URL remained the same?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it did. I had an idea to change the iso name (url here refers to name) to something general - like xenserver-tools and manage the same thing in actualIsoTemplate, but xs-tools.iso is used in some many places (specially in some test cases) which I ended up not changing this. Potentially it only needs a wiki/user guide doc only.

@@ -536,7 +536,7 @@ private void createXsToolsISO() {
id = _tmpltDao.getNextInSequence(Long.class, "id");
VMTemplateVO template =
VMTemplateVO.createPreHostIso(id, isoName, isoName, ImageFormat.ISO, true, true, TemplateType.PERHOST, null, null, true, 64, Account.ACCOUNT_ID_SYSTEM,
null, "xen-pv-drv-iso", false, 1, false, HypervisorType.XenServer);
null, "XenServer Tools Installer ISO (xen-pv-drv-iso)", false, 1, false, HypervisorType.XenServer);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will create an entry for deployments that do not have it yet using this new "standardized name". However, for environment that already had it from previous versions, the name (display_text) will not be updated, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is correct, I can explicitly add tmplt.setDisplayText("XenServer Tools Installer ISO (xen-pv-drv-iso)"); at line 544 (couple lines down here)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is the right thing to do, so we get consistency.
You have to save the tmplt as well right? So, you update the data in the database

@@ -2630,6 +2631,22 @@ public VDI getIsoVDIByURL(final Connection conn, final String vmName, final Stri
}
}

private String actualIsoTemplate(final Connection conn) throws BadServerResponse, XenAPIException, XmlRpcException {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind creating a documentation for the method? Then, you do not need comments in line.
Also, this is a good method to create a unit test.

Copy link
Contributor Author

@khos2ow khos2ow Jan 4, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add the documentation, but I really don't think unit testing on this specific method will be possible. Because it needs a live XenServer Host for the method to connect to (both Host.getByUuid(conn, _host.getUuid()); and host.getRecord(conn);) and I'm not sure if they can be Mocked in some way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need a real host. We only want to test the logic. Therefore, you can mock the method calls. PowerMockito and Mockito are there for this kind of job.

@blueorangutan
Copy link

Trillian test result (tid-2003)
Environment: xenserver-65sp1 (x2), Advanced Networking with Mgmt server 7
Total time taken: 44235 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2365-t2003-xenserver-65sp1.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 65 look OK, 2 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_02_edit_template Failure 90.15 test_templates.py
test_07_resize_fail Failure 61.21 test_volumes.py

@rohityadavcloud
Copy link
Member

@khos2ow can you address outstanding issues?

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1584

@khos2ow
Copy link
Contributor Author

khos2ow commented Jan 5, 2018

@rhtyd I don't think the failed tests are related to the change. The change was in attaching ISO. Can you re-run the test?

@rohityadavcloud
Copy link
Member

@blueorangutan package

@blueorangutan
Copy link

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan
Copy link

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1604

@rohityadavcloud
Copy link
Member

@khos2ow I've kicked another test round against xenserver 7.1

@blueorangutan
Copy link

Trillian test result (tid-2050)
Environment: xenserver-71 (x2), Advanced Networking with Mgmt server 7
Total time taken: 30548 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2365-t2050-xenserver-71.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_vm_iso.py
Intermitten failure detected: /marvin/tests/smoke/test_scale_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Smoke tests completed. 64 look OK, 3 have error(s)
Only failed tests results shown below:

Test Result Time (s) Test File
test_01_scale_vm Error 15.20 test_scale_vm.py
test_02_edit_template Failure 90.06 test_templates.py
test_07_resize_fail Failure 40.48 test_volumes.py

@rohityadavcloud
Copy link
Member

rohityadavcloud commented Jan 7, 2018

Test LGTM for 7.1 comparing baseline results against #2376. Test LGTM for XenServer 6.5sp1. The failures around template/volume are not related to this PR and also seen in #2376.
Merging this based on code reviews and test results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants