improve startup and overall performance

[Strubbl]

When I start XG it need some minutes (2 to 3 min) before the web interface is ready. And even when this interface is ready, it needs some more time to get all my saved searches and downloads loaded.
Maybe my db files are too big? But how can I clean them up? E.g. I don't need the stuff older than a week or a couple of days.

File sizes are e.g.:
$ ll insgesamt 48M -rw-r--r-- 1 strubbl users 2,3K 4. Apr 18:44 xg.config -rw-r--r-- 1 strubbl users 48M 4. Apr 18:47 xgobjects.db -rw-r--r-- 1 strubbl users 200K 4. Apr 18:49 xgsnapshots.db

Any other suggestions to improve performance of XG?

[lformella]

The start up speed is limited by sqlite. Maybe I will find another embedded database to use, but atm you can only speed up loading time by installing a separate database like mysql and create your own hibernate config. If you need help, I can post an example.
XG will vacuum the sqlite file on startup to compact its size and uses a logic to drop bots and packets which are offline some time. If you want to shrink the size of the sqlite file you could delete some unused servers or channels, of course.

I noticed an slow interface loading speed, too. But as I am using an old atom 1,6ghz / 1,5gb ram / win7 laptop I thought it might be an hardware performance issue. Which specs does your XG pc have?

[Strubbl]

It's a normal Desktop PC. Core i5 with some GHz, 16 GB RAM and I already moved those db files to my SSD drive.
As i now know it is sqlite (and no bin anymore like in past) i am going to have a look inside the database when i am back home.
Maybe i can see why it is so big.

[hrvstr]

I have the same problem, running XG v3.1.0.1 on Windows 8.1.
My specs:
AMD Phenom II (4x3Ghz)
4 GB DDR2 RAM

running basically nothing else than standard windows services and XG.
CPU activity jumps from 15-30% during XG startup.
Web interface takes 2-3 minutes to load up.

I have added 2 servers and 3 channels

[Strubbl]

Hmm, had a look into the database. 160k packets are in and are all up to date. So the size of the db is normal.

Maybe one could load the whole database into the RAM and then start xg using the RAM database? I don't have a clear idea about it atm, but maybe you have one?

I am going to think about it, too. Maybe a tmpfs or the like. A solution might be a script, manually copying the config files of XG to a tmpfs, starting XG and using the db out of this tmpfs. Shutdown with a script, too. This copies the db back to hard disk after shutdown. So, if PC crashes, database is still in place, but not up to date. Could that delete files in the XG/tmp folder? Any other side effects possible when using an old db?

What is the best solution to stop XG via console/script? Sending some signal with the kill command?
But in the end I have no idea how fast sqlite is when used from RAM.

[scottc]

Why must we vacuum the DB on startup? Could we periodically vacuum during runtime instead. for example every 5 minutes do a mini vacuum.

SELECT TOP 50 * FROM packages WHERE LastVacuumDate < NOW() - VacuumAgeThreshold

Or something like that, my SQL skills isn't the best. The top 50 and the 5 minutes could be adjusted to suit performace vs cleanness.

Loading the entire DB into ram is a bad idea, better off just creating indexes for the data you need to query frequently; like primary keys etc.

Could it be a threading issue? spawn a worker thread to handle the vacuuming, so the main app execution does not get blocked.

There are a few ways to deal with performance issues.
moving code out of the bottleneck
caching
indexing
optimizing the code (reducing checks via broadphases, replacing expensive methods, reducing overheads, consider impact on CPU, RAM, HDD performance and bandwidth between components etc)

Sorry, I should probably read the source code before offering suggestions.

[Strubbl]

I think Lars is switching the database from sqlite to db4o. So we can expect performance boost with that.

[lformella]

[adrenochrom]

lformella:
"The start up speed is limited by sqlite. Maybe I will find another embedded database to use, but atm you can only speed up loading time by installing a separate database like mysql and create your own hibernate config. If you need help, I can post an example."

Is it possible to post a short howto about changing the the database from sqlite to mysql?

[scottc]

https://github.com/scottc/xdcc-grabscher/commit/3c238611600ffd742872cacf6853f249f414dcba
Postsponing it to after the web interface plugin is loaded, and spawning a thread works wonders. Although it may cause a race condition, if you attempt to manipulate records which are just about to be deleted etc.

[lformella]

@adrenochrom
The next XG version will include another db engine which is still a flat file, but much faster than sqlite.

@scottc
The reset task is removed in the next XG version, because the relevant properties are now transient. The duplicate checker task is the most expensive and it should be safe to put it into a thread. But I think its better to run both file tasks before any plugin / database interaction.

I am keeping XG running untill it stops working or the machine reboots. So if the startup process takes two minutes or just 30 seconds is not really important. Do you restart XG so often that every second matters?

[scottc]

I run it as a service on boot, on my desktop machine that I shutdown everynight. I don't mind the current performance, but faster is better.

Here is a benchmark, for checking duplicates, comparing N^2 foreach loop, vs Linq groupby broadphase.
https://dotnetfiddle.net/EjRKKW

Number of objects: 5

N^2 00:00:00.0000023
BroadPhase 00:00:00.0000049
BroadPhase is faster by: -0.0026ms
BroadPhase is faster by: -72%

Number of objects: 50

N^2 00:00:00.0000827
BroadPhase 00:00:00.0000264
BroadPhase is faster by: 0.0563ms
BroadPhase is faster by: 103%

Number of objects: 500

N^2 00:00:00.0084707
BroadPhase 00:00:00.0001899
BroadPhase is faster by: 8.2808ms
BroadPhase is faster by: 191%

Number of objects: 5000

N^2 00:00:01.0426004
BroadPhase 00:00:00.0069732
BroadPhase is faster by: 1035.6272ms
BroadPhase is faster by: 197%

N^2, isn't actually that costly in this case, because we are just doing two equality comparisons.

Also we are firing many object removed events, which is calling the database Commit() method, for every object. We can probably commit deleting all the duplicates in one transaction. Work directly with db4o.

[Strubbl]

Can u please share ur service unit in the wiki?

[Strubbl]

Is there a plan for a next release available? Last is from May... or shall i better build the SW by myself?

[scottc]

Can u please share ur service unit in the wiki? - Strubbl

What I'm using, for unit file:

[Unit]
Description=xdcc grabscher
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/mono /home/scottc/Projects/xdcc-grabscher/XG.Application/bin/Release/XG.Application.exe

[Install]
WantedBy=multi-user.target

On archlinux:

# swap gedit for whatever your text editor of choice is (nano, vim, kate etc)
sudo gedit /etc/systemd/system/xdccgrabscher.service
# paste in the unit above, edit the ExecStart, to wherever you have XG installed and then save the file and close gedit.

# enable the service etc.
sudo systemctl status xdccgrabscher.service
sudo systemctl enable xdccgrabscher.service
sudo systemctl start xdccgrabscher.service

Ideally, I would install the app somewhere following the Linux Standards Base convention, or if i was going to keep the the app in my home directory, I might consider running the service under the service manager of my user, by using the systemctl --user flag.

Is there a plan for a next release available? Last is from May... or shall i better build the SW by myself?

I should consider maintaining two packages in the AUR, xdcc-grabscher and xdcc-grabscher-git.

[Strubbl]

If you could maintain a git verison in the AUR, I would like to install it! :) 👍

[Strubbl]

@lformella how about a new release?
@scottc have you started your activities for the AUR? Are any pkgbuilds already available?

[scottc]

@Strubbl xbuild does not like the Wix (windows installer) project, but the rest builds fine. Need to figure out a way to exlude building this project, or ignore the xbuild error.

What I've got so far.

# Maintainer: Scott Campbell <@.com>

_pkgname=xdcc-grabscher
_solution=XG.sln

pkgname=$_pkgname-git
pkgver=3.2.0.0.r8.392e95e
pkgrel=1
pkgdesc="A XDCC download client."
arch=('i686' 'x86_64')
url="https://github.com/lformella/xdcc-grabscher"
license=('GPL')
depends=('mono')
makedepends=('mono' 'git' 'nuget')
provides=("${pkgname%-*}")
conflicts=("${pkgname%-*}")
source=("git+${url}.git")
sha256sums=('SKIP') #autofill using updpkgsums

pkgver() {
  cd ${pkgname%-*}

  printf "%s" "$(git describe --tags | sed 's/^v//; s/-/.r/; s/-g/./')"
}

prepare() {
  cd "$_pkgname"
  nuget restore
}

build() {
  cd "$_pkgname"
  xbuild "$_solution" /p:Configuration=Release
}

package() {
  echo "do package stuff..."
  #cd "$pkgname-$pkgver"
}

[Strubbl]

@scottc Or maybe it is possible to wiki a manual how to build it with monodevelop or the like? Because I get the wix issues there, too.

[scottc]

@Strubbl See Issue #69, mono is expecting a target called "Build". I posted a fix as well, but I'm hoping to find a better solution.

[Strubbl]

Thanks, I am going to give a try.

[lformella]

@Strubbl there is a smartirc4net bug which forces an irc server with heavy load to close the connection after a few minutes (#47) - I will create the next release if this is solved

[lformella]

Take a look at the new version v3.3.0.0

[Strubbl]

Yeah, fast as the wind