Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory usage/leak #121

Closed
tacc-desantis opened this issue Mar 16, 2023 · 23 comments
Closed

Memory usage/leak #121

tacc-desantis opened this issue Mar 16, 2023 · 23 comments

Comments

@tacc-desantis
Copy link

Hello,

First and foremost, thank you for this software! It works incredibly well, and we're seeing significant differences querying the index file system (multiple PB's) vs. using the traditional find command.

I've run into an issue, unfortunately, in terms of memory usage during a re-index process. Basically, the gufi_dir2index process consumes too much memory leading to system instability. The test system in question has 256 GB of RAM available and the re-index has caused many OOM events.

Are there any controls and/or configuration options I can set on the GUFI side in order to not consume all available memory, leading to OOM events? For what it's worth, there doesn't seem to be a correlation between the number of files and/or space consumed on the file system that we're indexing. I built GUFI from the latest commit at the time, which was a8d9328.

Thank you!
John DeSantis
@dmanno
Copy link
Collaborator

dmanno commented Mar 20, 2023

Hi John,

We are actively looking into this. We will update progress here. If you could please share a bit of the details around the FS being indexed, that would be great. We are expecting that there are directories with very high amounts of sub-dirs, causing the queues to grow quickly. If you'd like to discuss via email, feel free to reach out to dmanno@lanl.gov

@tacc-desantis
Copy link
Author

Hello Dominic!

Thanks for the reply! Just for public visibility, the file system I'm trying to index is BeeGFS. It is 8.1 PB usable and currently has 1,034,075,745 inodes in use. I'll email you directly with additional details.

John DeSantis

@calccrypto
Copy link
Collaborator

calccrypto commented Mar 20, 2023 •
edited
Loading

Hi. Please try out these two branches:

insitu
gufi_dir2index -C <count> ..., where count is the maximum number of subdirectories in each directory that are enqueued for parallel processing. Any remaining subdirectories are processed in the current directory's thread, reducing the number of active allocations. The value that should be used will depend on the distribution of branching factors of the specific tree you are using.

deferred-queue
gufi_dir2index -M <bytes> ..., where bytes is a soft target memory size for the queues to attempt to reduce the overall memory footprint to. bytes is divided by the number of threads. Work is put on a separate per-thread queue that is not processed until the normal per-thread queue is emptied.

@tacc-desantis
Copy link
Author

Hello,

Thank you! I am running the deferred-queue build now. I'll update this issue with the results once it is done.

John DeSantis

@tacc-desantis
Copy link
Author

Hello,

I'm pleased to state that I've noticed stability in the memory footprint so far! I'll update again once the indexing finishes. Feel free to review the screen shot I've included to show that the deferred-queue branch has had a positive effect, including a small decrease in the amount of memory being used

Screenshot from 2023-03-21 16-43-39

And for what it's worth, the directory in question has 38,574,219 sub directories.

@tacc-desantis
Copy link
Author

Hello,

I wanted to provide an update. Unfortunately, indexing on two specific directories still continues to fail due to memory usage, despite either branch and various combinations of arguments, so we've reached out to the users in question to tidy up their directories.

Once we hear back from them I'll get a count of their current directory structures and verify items have been culled. I'll then re-run the indexing process (both methods) and report back.

Thanks again!
John DeSantis

@calccrypto
Copy link
Collaborator

Please give the reduce-work-size branch a try. The data structure that is enqueued has been reduced in size by over 40%. Additionally, if CMake detects the zlib library and -C (with no value this time) is passed to gufi_dir2index, the work items will be compressed.

@tacc-desantis
Copy link
Author

Hello,

The reduce-work-size branch is extremely promising! So far, not only has the indexing process run longer (4x) for this specific directory, it's memory footprint has been reduced by ~83%!!!! Please see the screen shot below to see the fruits of your labor.

Screenshot from 2023-03-30 09-56-19

I'll update again once the process completes or is terminated.

@tacc-desantis
Copy link
Author

Hello,

Both indexes completed without any logged errors. CPU and memory statistics for both processes is listed below:

56,591,355 directories & 80% reduction in memory footprint:

23962.18user 9703.66system 35:46:25elapsed 26%CPU (0avgtext+0avgdata 41947132maxresident)k 104162440inputs+3706836056outputs (24major+44473160minor)pagefaults 0swaps

38,574,219 directories & 91.5% reduction in memory footprint:

9857.33user 3419.66system 14:54:09elapsed 24%CPU (0avgtext+0avgdata 18088904maxresident)k 30993064inputs+1370928848outputs (2major+25345753minor)pagefaults 0swaps

We will need to provide more storage for the out of tree index though, as these two directories consumed all 3.7 TB allocated for the indexes.

Thank you!
John DeSantis

@garygrider
Copy link
Contributor

garygrider commented Apr 2, 2023 via email

@tacc-desantis
Copy link
Author

Hello,

Thanks so much for the challenging use case to help make the software better.

Certainly! I thoroughly appreciate the time and fixes the development team produced.

Potentially these things could help you and of course it would be nice to know if these tools
can handle such a wide directory structure that your site has, something we could help ensure works.

Agreed. Currently, I've only been using gufi_find for querying the tree, so I'll need to read-up on available documentation (including recent slides) and look at some of the commit logs to take advantage of these options. Despite this, gufi_find is still a gem, as our current production need is to speed up file system purging. Once we have developed a proper workflow for this process, we'll most likely apply it to several other production file systems, and then begin to utilize GUFI's other features for richer file system accounting.

If there are any other features the development team would like to test against large, production file systems, let me know.

Thank you again!
John DeSantis

@garygrider
Copy link
Contributor

garygrider commented Apr 3, 2023 via email

@calccrypto
Copy link
Collaborator

The changes made in the 3 branches have been merged into one branch and then merged with master via #125. Compression with zlib has been changed to use -e since -C was used by the insitu branch.

Please let us know if the combining of all of the changes broke something.

@tacc-desantis
Copy link
Author

Hello,

The changes made in the 3 branches have been merged into one branch and then merged with master via #125. Compression with zlib has been changed to use -e since -C was used by the insitu branch.

Excellent!

Please let us know if the combining of all of the changes broke something.

Will do! I am running an index now. Also, I've deliberately have skipped the indexing of the problematic directories now since users are actively removing content; one of the logs had 1k+ lines stating that directories couldn't be opened due to the active file/directory removals.

Thanks again for such amazing responses from the development team!

John DeSantis

@tacc-desantis
Copy link
Author

Gary,

Glad it is helping you. Yes guf_find is about 1/50th of the capability gufi has.

Where's that "mind blown" gif???

Very complex things are pretty easy like make me a sorted histogram of all the file types in the entire system to simple things like find me the top 3 largest directories. Hope you find use for some of it…

Thanks for these suggestions, a seed has been planted...

John DeSantis

@tacc-desantis
Copy link
Author

Hello,

Marking this as resolved! I am no longer seeing memory issues that I had seen previously. Thanks again for all of the development team's hard work.

John DeSantis

@tacc-desantis
Copy link
Author

tacc-desantis commented Apr 6, 2023 via email •
edited
Loading

@garygrider
Copy link
Contributor

garygrider commented Apr 6, 2023 via email

@tacc-desantis
Copy link
Author

Gary,

Try
./gufi_query -n 1 -E "select path(uid,uid),* from entries;" /gufi_index/lo_scemo/ | head -10

And of course
./gufi_query -n 1 -E ‘select path(uid,uid) || “\” || name, size, uid, uidtouser(uid,20) from entries;" /gufi_index/lo_scemo/ | head -10

These work perfectly, thank you for the insight!

We need to produce a “sql guide go gufi”
I imagine we will because we have a tutorial coming up at MSST in May

A guide would be most beneficial, especially since (as you already said) there are a lot more data points available via gufi_query vs. the wrapper tools gufi_find, gufi_ls, etc. I'd love to attend MSST, but it's too short of a notice and I'm in the middle of a cross country move :(

You can look at the entire schema in the code or just go to a dir and do
Sqlite3 dbname where dbname is the name of the database in that dir
Then something like .schema or something

Agreed, that's what I did since we all love to abuse and exploit SQL!

There is a LOT there and happy to describe if it helps

You may want to rescind that offer, hahaha!

Thanks again,
John DeSantis

@calccrypto
Copy link
Collaborator

calccrypto commented Apr 6, 2023 •
edited
Loading

I pushed some commits that restore the original path(), epath(), and fpath() sqlite functions. path() no longer takes in arguments. I have renamed the function with 2 arguments (summary.name and summary.rollupscore) to rpath(), which is meant for handling path names in rolled up indices. Its intended use in the -E query is:

SELECT rpath(summary.name, summary.rollupscore) || "/" || pentries.name
FROM summary, pentries
WHERE summary.inode == pentries.pinode;

However, because we are not in the business of forcing user queries to be in a certain format, it is easy to misuse rpath().

uidtouser, gidtogroup, and several other functions had an extra unexpected argument. They have been removed. The extra argument was to help with alignment of output, but they were always unused and set to 0.

@garygrider
Copy link
Contributor

garygrider commented Apr 6, 2023 via email

@garygrider
Copy link
Contributor

garygrider commented Apr 7, 2023 via email

@calccrypto
Copy link
Collaborator

@tacc-desantis I just pushed GUFI-SQL.docx, Gary's guide on using GUFI's SQL schema and functions, to the docs directory. Please pull the latest version of GUFI to get all of the features mentioned in the guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@dmanno @calccrypto @garygrider @tacc-desantis