Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

inter-session starvation issue, particularly affects sys.queue #3665

Closed
monetdb-team opened this issue Nov 30, 2020 · 0 comments
Closed

inter-session starvation issue, particularly affects sys.queue #3665

monetdb-team opened this issue Nov 30, 2020 · 0 comments

Comments

@monetdb-team
Copy link

@monetdb-team monetdb-team commented Nov 30, 2020

Date: 2015-02-05 08:02:39 +0100
From: sorear
To: MonetDB5 devs <>
Version: 11.19.7 (Oct2014-SP1)
CC: @njnes

Last updated: 2015-05-07 12:37:28 +0200

Comment 20630

Date: 2015-02-05 08:02:39 +0100
From: sorear

User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.94 Safari/537.36
Build Identifier:

If I run a long query in one terminal session, and a very short one in another, MonetDB seems to wait for the long query to finish before running the short query. In particular, I can't get useful results from sys.queue() because by the time sys.queue starts, it's the only running query and reports only on itself(?).

Reproducible: Always

Steps to Reproduce:

  1. start two mclients in different terminals. In the first:

Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2014-SP1)
Database: MonetDB v11.19.7 (Oct2014-SP1), '...'
Type \q to quit, ? for a list of available commands
auto commit mode: on
sql>create table blah as select * from sys.generate_series(10000,13000) with data;
operation successful (2.098ms)

second:

Welcome to mclient, the MonetDB/SQL interactive terminal (Oct2014-SP1)
Database: MonetDB v11.19.7 (Oct2014-SP1), 'mapi:monetdb://sorear1.dev.gudtech.com:50000/gt'
Type \q to quit, ? for a list of available commands
auto commit mode: on

first, issue a long-running command:

sql>select count(distinct a.value * b.value) from blah a, blah b;

second attempt to see this command:

sql>select * from sys.queue();

several seconds later, this appears in the first:

+---------+
| L1 |
+=========+
| 4072685 |
+---------+
1 tuple (9.0s)

simultaneously in the second:

+------+---------+----------------------------+----------------------------+----------+---------+------------+----------------------------+
| qtag | user | started | estimate | progress | status | tag | query |
+======+=========+============================+============================+==========+=========+============+============================+
| 40 | monetdb | 2015-02-05 06:55:21.000000 | null | null | running | 12783204@0 | select * from sys.queue(); |
+------+---------+----------------------------+----------------------------+----------+---------+------------+----------------------------+
1 tuple (2.5s)

when not busy, queue requests finish instantly:

sql>select * from sys.queue();
+------+---------+----------------------------+----------------------------+----------+---------+------------+----------------------------+
| qtag | user | started | estimate | progress | status | tag | query |
+======+=========+============================+============================+==========+=========+============+============================+
| 41 | monetdb | 2015-02-05 06:55:24.000000 | null | null | running | 12783204@0 | select * from sys.queue(); |
+------+---------+----------------------------+----------------------------+----------+---------+------------+----------------------------+
1 tuple (0.638ms)

Expected Results:

I'm not certain this is truly a bug, but it'd be very very useful if I could observe concurrently running queries with select * from sys.queue() or something functionally equivalent.

Comment 20631

Date: 2015-02-05 08:04:52 +0100
From: sorear

This particular example is on a VM with 2 cores; I suspect that makes a difference here.

Comment 20714

Date: 2015-03-13 14:08:52 +0100
From: @sjoerdmullender

The problem is caused by changeset e79b2edeba71 from January 2014.

That changeset introduced a semaphore to limit concurrency. But this was done in such a way that no new queries enter the fray if the maximum level of concurrency (currently set at half the number of cores) is reached.

Comment 20715

Date: 2015-03-13 14:34:24 +0100
From: @sjoerdmullender

(In reply to comment 2)

The problem is caused by changeset e79b2edeba71 from January 2014.

Make that Jan 2013.

Comment 20716

Date: 2015-03-13 18:14:32 +0100
From: sorear

Interesting. We've been seeing behavior on a couple of 16 core production machines where the system is under sustained MonetDB load (mserver5 consistently > 500% CPU), but when I run "select * from sys.queue" it waits several seconds and then reports that the select is the only thing running. It sounds like the example I included may not have been an accurate reproduction then?

Comment 20717

Date: 2015-03-16 14:51:31 +0100
From: MonetDB Mercurial Repository <>

Changeset 68156d38a8c4 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see http//devmonetdborg/hg/MonetDB?cmd=changeset;node=68156d38a8c4

Changeset description:

Don't limit incoming queries based on number of cores.
This fixes bug #3665.

Comment 20794

Date: 2015-04-14 12:35:56 +0200
From: @sjoerdmullender

The limiting semaphore has been removed. The problem should not occur anymore.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant