Change default Settings to virtio-scsi by default? (for fstrim support )

[4920441]

Hi,
how can I change the default parameters, so "fstrim" compatible vms are created by default without manually changing the xml files?
I found a couple of parameters, but not all I need to achieve this...:

<controller type='scsi' index='0' model='virtio-scsi'/>
<disk type='file' device='disk'>
 <driver name='qemu' type='qcow2' discard='unmap'/>
 <source file='/var/lib/libvirt/images/example.qcow2'/>
 <backingStore/>
 <target dev='sda' bus='scsi'/>
</disk>

Thanks for you help (again:-)

Cheers,

4920441

[catborise]

hi,
Doesn't "INSTANCE_VOLUME_DEFAULT_BUS = 'virtio'" key value in settings.py file solve the problem?

INSTANCE_VOLUME_DEFAULT_BUS = 'virtio' -> INSTANCE_VOLUME_DEFAULT_BUS = 'scsi' creates the xml like you specified.

<controller type='scsi' index='0' model='virtio-scsi'>
     <alias name='scsi0'/>
     <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
   </controller>
<disk type='file' device='disk'>
     <driver name='qemu' type='qcow2' cache='directsync'/>
     <source file='/var/lib/libvirt/images/t1.qcow2'/>
     <backingStore/>
     <target dev='sdb' bus='scsi'/>
     <alias name='scsi0-0-0-1'/>
     <address type='drive' controller='0' bus='0' target='0' unit='1'/>
   </disk>

or you want different thing?

[4920441]

Nearly but the discard key is still missing....

<driver name='qemu' type='qcow2' discard='unmap'/>

[catborise]

i am working on create instance . it is a big change. i will consider it with the new change.

[catborise]

@4920441 hi, can you please checkout my repo to test new vm creation operation and discard etc adding. please check and add new settings from settings.py.template

[4920441]

Hi catborise,

yeah sure, I'll do that. I come back to you.

Cheers,

4920441

[4920441]

regarding to you upgrade description, there are some problems now, which were not there when I first installed it last week on this server...directly when tryring to update:

root@ryzenserver:/srv/webvirtcloud# virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /srv/webvirtcloud/venv/bin/python2
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/virtualenv.py", line 2379, in
main()
File "/usr/lib/python3/dist-packages/virtualenv.py", line 724, in main
symlink=options.symlink)
File "/usr/lib/python3/dist-packages/virtualenv.py", line 950, in create_environment
site_packages=site_packages, clear=clear, symlink=symlink))
File "/usr/lib/python3/dist-packages/virtualenv.py", line 1282, in install_python
shutil.copyfile(executable, py_executable)
File "/usr/lib/python2.7/shutil.py", line 97, in copyfile
with open(dst, 'wb') as fdst:
IOError: [Errno 26] Text file busy: '/srv/webvirtcloud/venv/bin/python2'
root@ryzenserver:/srv/webvirtcloud# ^C venv/bin/
root@ryzenserver:/srv/webvirtcloud# ls
accounts create Dockerfile manage.py static vrtManager
computes datasource gunicorn.conf.py networks storages webvirtcloud
conf db.sqlite3 instances nwfilters templates
_config.yml dev interfaces README.md Vagrantfile
console doc logs secrets venv
root@ryzenserver:/srv/webvirtcloud# cd webvirtcloud/
root@ryzenserver:/srv/webvirtcloud/webvirtcloud# virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in /srv/webvirtcloud/webvirtcloud/venv/bin/python2
Also creating executable in /srv/webvirtcloud/webvirtcloud/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
root@ryzenserver:/srv/webvirtcloud/webvirtcloud# source venv/bin/activate
(venv) root@ryzenserver:/srv/webvirtcloud/webvirtcloud# git pull
Bereits aktuell.
(venv) root@ryzenserver:/srv/webvirtcloud/webvirtcloud# pip install -U -r conf/requirements.txt
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
ERROR: Could not open requirements file: [Errno 2] Datei oder Verzeichnis nicht gefunden: 'conf/requirements.txt'
(venv) root@ryzenserver:/srv/webvirtcloud/webvirtcloud# cd ..
(venv) root@ryzenserver:/srv/webvirtcloud# pip install -U -r conf/requirements.txt
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting Django==1.11.25
Using cached https://files.pythonhosted.org/packages/3c/ed/06a81a65fa00f766f2dbda94d09e946aa65c23e6d7ca3532984627a6c75a/Django-1.11.25-py2.py3-none-any.whl
Processing /root/.cache/pip/wheels/c4/0a/38/9bdfe79aeece868708627b1d891aef2813404ee9739b0a4b85/websockify-0.9.0-cp27-none-any.whl
Collecting gunicorn==19.9.0
Using cached https://files.pythonhosted.org/packages/8c/da/b8dd8deb741bff556db53902d4706774c8e1e67265f69528c14c003644e6/gunicorn-19.9.0-py2.py3-none-any.whl
Collecting lxml==4.2.5
Using cached https://files.pythonhosted.org/packages/e5/14/f4343239f955442da9da1919a99f7311bc5627522741bada61b2349c8def/lxml-4.2.5-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pytz
Using cached https://files.pythonhosted.org/packages/e7/f9/f0b53f88060247251bf481fa6ea62cd0d25bf1b11a87888e53ce5b7c8ad2/pytz-2019.3-py2.py3-none-any.whl
Processing /root/.cache/pip/wheels/f6/96/55/ee5e72c5e8d4b238f1bc354d2fa999083c1eab7c69a14ea3e2/rwlock-0.0.7-cp27-none-any.whl
Collecting numpy
Using cached https://files.pythonhosted.org/packages/d7/b1/3367ea1f372957f97a6752ec725b87886e12af1415216feec9067e31df70/numpy-1.16.5-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: pytz, Django, numpy, websockify, gunicorn, lxml, rwlock
Successfully installed Django-1.11.25 gunicorn-19.9.0 lxml-4.2.5 numpy-1.16.5 pytz-2019.3 rwlock-0.0.7 websockify-0.9.0
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 327, in execute
self.check()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check```

include_deployment_checks=include_deployment_checks,

File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 62, in _run_checks
issues.extend(super(Command, self)._run_checks(**kwargs))
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
return checks.run_checks(**kwargs)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
new_errors = check(app_configs=app_configs)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config
return check_resolver(resolver)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver
return check_method()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 256, in check
for pattern in self.url_patterns:
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 407, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 400, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/srv/webvirtcloud/webvirtcloud/urls.py", line 3, in
from instances.views import index
File "/srv/webvirtcloud/instances/views.py", line 19, in
from vrtManager.hostdetails import wvmHostDetails
File "/srv/webvirtcloud/vrtManager/hostdetails.py", line 2, in
from vrtManager.connection import wvmConnect
File "/srv/webvirtcloud/vrtManager/connection.py", line 1, in
import libvirt
ImportError: No module named libvirt
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py migrate^C
Unknown command: 'migrate\x03'
Type 'manage.py help' for usage.
(venv) root@ryzenserver:/srv/webvirtcloud# pip install -U -r conf/requirements.txt
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Requirement already up-to-date: Django==1.11.25 in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 1)) (1.11.25)
Requirement already up-to-date: websockify==0.9.0 in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 2)) (0.9.0)
Requirement already up-to-date: gunicorn==19.9.0 in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 3)) (19.9.0)
Requirement already up-to-date: lxml==4.2.5 in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 4)) (4.2.5)
Requirement already up-to-date: pytz in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 6)) (2019.3)
Requirement already up-to-date: rwlock in ./webvirtcloud/venv/lib/python2.7/site-packages (from -r conf/requirements.txt (line 7)) (0.0.7)
Requirement already satisfied, skipping upgrade: numpy in ./webvirtcloud/venv/lib/python2.7/site-packages (from websockify==0.9.0->-r conf/requirements.txt (line 2)) (1.16.5)
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py migrate
Traceback (most recent call last):
File "manage.py", line 10, in
execute_from_command_line(sys.argv)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 364, in execute_from_command_line
utility.execute()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/init.py", line 356, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
self.execute(*args, **cmd_options)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 327, in execute
self.check()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 359, in check
include_deployment_checks=include_deployment_checks,
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 62, in _run_checks
issues.extend(super(Command, self)._run_checks(**kwargs))
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/management/base.py", line 346, in _run_checks
return checks.run_checks(**kwargs)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/registry.py", line 81, in run_checks
new_errors = check(app_configs=app_configs)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 16, in check_url_config
return check_resolver(resolver)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/core/checks/urls.py", line 26, in check_resolver
return check_method()
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 256, in check
for pattern in self.url_patterns:
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 407, in url_patterns
patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/utils/functional.py", line 35, in get
res = instance.dict[self.name] = self.func(instance)
File "/srv/webvirtcloud/webvirtcloud/venv/local/lib/python2.7/site-packages/django/urls/resolvers.py", line 400, in urlconf_module
return import_module(self.urlconf_name)
File "/usr/lib/python2.7/importlib/init.py", line 37, in import_module
import(name)
File "/srv/webvirtcloud/webvirtcloud/urls.py", line 3, in
from instances.views import index
File "/srv/webvirtcloud/instances/views.py", line 19, in
from vrtManager.hostdetails import wvmHostDetails
File "/srv/webvirtcloud/vrtManager/hostdetails.py", line 2, in
from vrtManager.connection import wvmConnect
File "/srv/webvirtcloud/vrtManager/connection.py", line 1, in
import libvirt
ImportError: No module named libvirt
(venv) root@ryzenserver:/srv/webvirtcloud#

[catborise]

sudo virtualenv venv statement causes that problem , do not use it... because default python version is python3 on your environment.
for now app works with python2.

[4920441]

Thanks, now it ran without an error despite :

root@ryzenserver:/srv/webvirtcloud# python manage.py migrate
Operations to perform:
  Apply all migrations: accounts, admin, auth, computes, contenttypes, create, instances, logs, sessions
Running migrations:
  No migrations to apply.
  Your models have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py makemigrations
Migrations for 'computes':
  computes/migrations/0003_auto_20191214_0942.py
    - Alter field hostname on compute
    - Alter field name on compute
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py migrate
Operations to perform:
  Apply all migrations: accounts, admin, auth, computes, contenttypes, create, instances, logs, sessions
Running migrations:
  Applying computes.0003_auto_20191214_0942... OK
(venv) root@ryzenserver:/srv/webvirtcloud# service supervisor restart
(venv) root@ryzenserver:/srv/webvirtcloud# exit

But in

webvirtcloud/settings.py.template

are no other options as before, I think.....

I did adapt those befor myself:

 INSTANCE_VOLUME_DEFAULT_FORMAT = 'qcow2'
 #INSTANCE_VOLUME_DEFAULT_BUS = 'virtio'
 INSTANCE_VOLUME_DEFAULT_BUS = 'scsi'
 INSTANCE_VOLUME_DEFAULT_CACHE = 'directsync'
 # up to os, 0=root, 107=qemu or libvirt-bin(for ubuntu)
 INSTANCE_VOLUME_DEFAULT_OWNER = {'uid': 0, 'guid': 0}
 

[catborise]

you can use your settings as before. ..
with this update there are some new settings. you can use them as you wish

new entries

# Volume optionals: two variable: disk driver type is file and network(rbd, iscsi),
# optionals : discard='unmap|ignore', detect_zeroes='on|off|unmap', copy_on_read='on|off'
# Example: {"file": "discard='unmap' copy_on_read='on'", "network": "detect_zeroes='unmap'"}
INSTANCE_VOLUME_DEFAULT_DRIVER_OPTS = {"file": "", "network": ""}

# Cpu modes: host-model, host-passthrough, custom
INSTANCE_CPU_DEFAULT_MODE = 'host-model'

# Chipset/Machine: pc or q35 for x86_64
INSTANCE_MACHINE_DEFAULT_TYPE = `'q35'

[4920441]

Hi,
after adding the new variables and restarted the hypervisor I only get a

502 Bad Gateway
nginx/1.16.1

....
Even when I comment the lines out again, it does not come back.... Don't see why at the moment....

# available: default (grouped), nongrouped
VIEW_INSTANCES_LIST_STYLE = 'grouped'

INSTANCE_VOLUME_DEFAULT_FORMAT = 'qcow2'
#INSTANCE_VOLUME_DEFAULT_BUS = 'virtio'
INSTANCE_VOLUME_DEFAULT_BUS = 'scsi'
INSTANCE_VOLUME_DEFAULT_CACHE = 'directsync'
# up to os, 0=root, 107=qemu or libvirt-bin(for ubuntu)
INSTANCE_VOLUME_DEFAULT_OWNER = {'uid': 0, 'guid': 0}

# Volume optionals: two variable: disk driver type is file and network(rbd, iscsi),
# optionals : discard='unmap|ignore', detect_zeroes='on|off|unmap', copy_on_read='on|off'
# Example: {"file": "discard='unmap' copy_on_read='on'", "network": "detect_zeroes='unmap'"}i
#INSTANCE_VOLUME_DEFAULT_DRIVER_OPTS = {"file": "discard='unmap'"}
#, "network": ""}

# Cpu modes: host-model, host-passthrough, custom
#INSTANCE_CPU_DEFAULT_MODE = 'host-model'

# Chipset/Machine: pc or q35 for x86_64
#INSTANCE_MACHINE_DEFAULT_TYPE = `'q35'

2019/12/14 15:23:03 [error] 2726#2726: *42 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.64.50, server: , request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8000/", host: "192.168.64.10"
2019/12/14 15:23:03 [error] 2726#2726: *42 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.64.50, server: , request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8000/favicon.ico", host: "192.168.64.10", referrer: "http://192.168.64.10/"

[catborise]

your nginx working but not gunicorn.
do not make them comment line. app does not work if it cannot reach them.

check it to see if the services is working or not.
supervisorctl status

supervisorctl status
gstfsd                           RUNNING   pid 15298, uptime 2:26:06
novncd                           RUNNING   pid 15297, uptime 2:26:06
webvirtcloud                     RUNNING   pid 15296, uptime 2:26:06

[4920441]

It is not working any more but why? I doubt it has anything to to with editing the files, because I rolled that part completely back....:

supervisorctl status
novncd RUNNING pid 3022, uptime 0:00:17
webvirtcloud RUNNING pid 3030, uptime 0:00:17

[catborise]

stop supervisorctl services with supervisorctl stop webvirtcloud
then start it with manually to see if it working.

go to webvirtcloud directory

source vevn/bin/activate
python manage.py runserver

check outputs. and try to enter url with browser

[4920441]

Erm....
before I tried what you just wrote I did a

root@ryzenserver:/usr/local/bin# ls
gstfsd
root@ryzenserver:/usr/local/bin# python gstfsd
  File "gstfsd", line 45
    gfs.chmod(0700, "/root/.ssh")
                 ^
SyntaxError: invalid token
root@ryzenserver:/usr/local/bin#

and then:

root@ryzenserver:/srv/webvirtcloud# source venv/bin/activate
(venv) root@ryzenserver:/srv/webvirtcloud# python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
December 14, 2019 - 14:42:37
Django version 1.11.25, using settings 'webvirtcloud.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.
Error: That port is already in use.
(venv) root@ryzenserver:/srv/webvirtcloud# netstat -tulpen | grep 8000
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      33         38926      7215/python2        
(venv) root@ryzenserver:/srv/webvirtcloud# 

but anyway the webvirtcloud interface is reachable again and at least seems to be working again.

Where is it started automatically? An how did I start it the way I did - I mean, the first one was an erro r and the second one complained port 8000 is already used - so despite the first error it seems to have started at the first point....

But anyway:

(venv) root@ryzenserver:/srv/webvirtcloud# supervisorctl status
novncd                           RUNNING   pid 7214, uptime 0:13:50
webvirtcloud                     RUNNING   pid 7215, uptime 0:13:50
(venv) root@ryzenserver:/srv/webvirtcloud# 

[4920441]

After a reboot it's still working.... but with supervisorctl status it is still not shown as it is in your output:

Last login: Sat Dec 14 15:37:52 2019 from 192.168.64.50

root@ryzenserver: supervisorctl status
novncd                           RUNNING   pid 2944, uptime 0:00:08
webvirtcloud                     RUNNING   pid 2947, uptime 0:00:08

root@ryzenserver: netstat -tulpen | grep 8000
tcp        0      0 127.0.0.1:8000          0.0.0.0:*               LISTEN      33         33398      2947/python2