Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

large virtual memory spike on BLOB column COUNT #6498

Closed
monetdb-team opened this issue Nov 30, 2020 · 0 comments
Closed

large virtual memory spike on BLOB column COUNT #6498

monetdb-team opened this issue Nov 30, 2020 · 0 comments
Labels

Comments

@monetdb-team
Copy link

@monetdb-team monetdb-team commented Nov 30, 2020

Date: 2017-12-14 20:33:45 +0100
From: Anton Kravchenko <<kravchenko.anton86>>
To: SQL devs <>
Version: 11.27.11 (Jul2017-SP3)

Last updated: 2018-02-12 16:12:09 +0100

Comment 26008

Date: 2017-12-14 20:33:45 +0100
From: Anton Kravchenko <<kravchenko.anton86>>

User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36
Build Identifier:

'select count(v_blob) from blob_table' produces a virtual memory spike ~800Gb
[on the second run within same session]

Reproducible: Always

Steps to Reproduce:

Steps to Reproduce:

  1. to generate data by using Python
    hex_blob='47'*2000
    hex_blob_chunk=(hex_blob+'\n')*1000

f=open('blob_data.txt','w')
for ichunk in range(10000):
f.write(hex_blob_chunk)
f.close()

  1. to load data into Monet blob_table
    create table blob_table(v_blob blob);
    COPY 10000000 RECORDS INTO blob_table FROM '/home/blob_data.txt' using
    delimiters ',','\n' NULL AS '' LOCKED;

select count(v_blob) from blob_table; 1st run: OK
select count(v_blob) from blob_table; 2nd run: very slow [~30s] and memory spike of ~800 GB

Actual Results:

virtual memory spike of ~800GB

Expected Results:

virtual memory of ~25GB [MonetDB storage size of the blob column]

MonetDB 5 server v11.27.11 "Jul2017-SP3" (64-bit, 128-bit integers)
Copyright (c) 1993-July 2008 CWI
Copyright (c) August 2008-2017 MonetDB B.V., all rights reserved
Visit https://www.monetdb.org/ for further information
Found 503.6GiB available memory, 32 available cpu cores
Libraries:
libpcre: 8.32 2012-11-30 (compiled with 8.32)
openssl: OpenSSL 1.0.2k 26 Jan 2017 (compiled with OpenSSL 1.0.2l 25 May 2017)
libxml2: 2.9.1 (compiled with 2.9.1)
Compiled by: akravchenko@cent7-1 (x86_64-pc-linux-gnu)
Compilation: gcc -O3 -fomit-frame-pointer -pipe -D_FORTIFY_SOURCE=2
Linking : /usr/bin/ld -m elf_x86_64

Comment 26009

Date: 2017-12-15 13:25:57 +0100
From: MonetDB Mercurial Repository <>

Changeset a2f70a576335 made by Sjoerd Mullender sjoerd@acm.org in the MonetDB repo, refers to this bug.

For complete details, see https//devmonetdborg/hg/MonetDB?cmd=changeset;node=a2f70a576335

Changeset description:

Save no_mitosis flag in query cache.
This should fix bug #6498.

Comment 26010

Date: 2017-12-15 13:30:07 +0100
From: @sjoerdmullender

I couldn't really reproduce the big slow down, nor did I see the large memory spike, but I did find something that could still explain your observations.

We save query plans in a cache. The first query that you did was executed and saved, the second one was retrieved from the cache, but the optimizations that happened on the first and second queries were different (after the plan was stored into / retrieved from the cache). I have now fixed this.

Asuming this was indeed the problem, I'm closing this bug. Please reopen if this doesn't fix the problem.

Comment 26014

Date: 2017-12-15 18:59:42 +0100
From: Anton Kravchenko <<kravchenko.anton86>>

thanks Sjoerd, it works as expected now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
1 participant