-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Q: Memory consumption when using in-memory/internal mode #54
Comments
Goiardi's known to eat up a lot of memory in at least some configurations, although that's more than I would expect with only six clients. A couple of questions, though, just to get things figured out:
The size of the search index has been a definite problem in the past, although it certainly should be better than that with six clients. For a more immediate solution, since you're running it in docker anyway, have you considered running goiardi with postgres and the postgres search? |
1.) no Here's my setup shortly after a reboot of the host/restart of the containers:
|
In re: 3, the command is indeed |
I have now ~20 nodes and goiardi takes 10+ GB of RAM, CPU usage is very high (spikes). Also nginx is barking about some timed out connections to goiardi.
Looks like the number of pids grows, too. |
Bizarrely, the in-mem index is only using 22.7MB RAM with 50 nodes generated with fauxhai on OS X. This is... odd; I'm pretty sure I've also done this test with goiardi on Linux (I had to make some changes a long time ago to get the index more under control because some folks noticed it was taking up a lot of RAM), but it sure feels like there might be something different going on between OS X and Linux I hadn't noticed before. One question: did you build the binaries yourself, get the packages from packagecloud.io, or get one of the individual binaries from github? |
I'm using the following Dockerfile:
The one thing that is different from regular setups is that my chef-clients don't report all attributes to the server, maybe some attribute that goiardi expects are missing and cause those issues? typcial client.rb looks like: automatic_attribute_whitelist ["fqdn", "os", "os_version", "ohai_time", "hostname", "keys/ssh", "roles", "recipes", "ipaddress", "ip6address", "platform", "platform_version", "cloud/", "cloud_v2/", "chef_packages", "pspdfkit_docker", "status", "uptime", "uptime_seconds", "node_name", "plattform", "plattform_family", "plattform_version", "os", "os_version", "ohai/"]
chef_server_url "https://...chef-server.../"
default_attribute_whitelist []
encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"
log_location "/var/log/chef/client.log"
override_attribute_whitelist []
validation_client_name "chef-validator"
verify_api_cert true
# Using default node name (fqdn)
# Do not crash if a handler is missing / not installed yet
begin
rescue NameError => e
Chef::Log.error e
end |
arround using 16GB of RAM and gob is not able to persist anymore:
A restart now loads the old data. Data directory is filled with Unfortunately in a docker setup pprof is not reachable due to the typical default networking:
I suggest to allow a config option to whitelist a subnet/CIDR for debugging. Update 1 When goiardi runs in Update 2 I was able to get a heap dump but I was not able to get any useful information out of it (sorry, Go noob here) I've copied the data and index files to my macOS workstation and was able to export it (700MB in JSON), however an import of it with empty data and index files fails:
Update 3 $ for i in $(knife node list); do knife node show -l -F json $i > $i.json; done
$ ls -1 *.json|wc -l
27
$ du -hc *.json
12K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
40K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
12K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
24K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
20K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
12K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
80K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
16K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
12K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
24K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
36K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
60K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
36K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
8,0K xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.json
496K total Theory I suspect a leak in the gob/memory data store code that gets persisted and reloaded over time. I'll setup a new instance with the same data but the postgres backend, maybe it confirms that suspicion |
I was just thinking this morning that there might be a problem with the memory data store. I'm still working on duplicating your problem, but I suspect it's the problem. |
I've just moved the setup to postgres and resource consumption is amazingly small (same data/nodes/cookbooks).
IIRC even after the initial import into the in-memory store, the total memory usage was higher than the combination of goiardi and pg at this point 😲 |
I've pushed up a branch with some helpful changes for this at https://github.com/ctdk/goiardi/tree/mem-bonkers-tmp. It adds an argument to whitelist IPs for Looking at the pprof output, I think I've found some good candidates for the problem. I believe it's something to do with the data store (which you already pointed out), but I have a suspicion that it's something to do with the event log specifically. I haven't used the event log much with the in-memory mode, except to make sure it appeared to work. (When I do use it it's with Postgres, generally, since that's the main mode I use.) One question with moving that setup to postgres: are you still using the in-mem search, or is that using the postgres search? |
I switched to pg search:
|
That's cool (and definitely what I would do - I've found the pg search to be superior); I asked because I was curious if the mem usage you were reporting included the search or not. |
I've made progress on fixing the search query parser do deal with the negated range queries too, but am still being pulled in 800 directions all the time. :-/ I'm getting closer to getting them all wrapped up at least. |
We don't use complicated search terms. Usually just to search for data bags and nodes (based on "positive" assertions of fqdn, tags, roles or attributes). I like to keep things as simple as possible because most complicated chef setups tend to fail (or end in an unmaintainable rabbit hole) - at least this is my experience after ~7 years. |
No argument from me there -- when I was refactoring the search to fix the broken NOT queries, I looked at NOT range and both a) could find no evidence it was used anywhere, at least with my cursory glance & against chef-pedant and b) it seemed iffy if Lucene/Solr even supported that particular query*. Anecdotally, upwards of 95% of chef search queries are extremely basic. I've never felt that Solr was the right choice for chef search (like, it's both massive overkill and not particularly well suited to this kind of indexing), which was my motivation for coming up with a couple of replacements. Still, I'm only one guy with other things going on as well, and I'll admit that I've kind of neglected the in-memory aspect of goiardi in favor of the postgres part. I've even been considering dropping MySQL from 1.0.0 when that's ready, but I haven't decided yet.
|
So… I've no an issue with the pg search ;) Ohai by default provides a couple of automatic attributes per node, e.g. {
"cloud_v2": {
"provider": "digital_ocean"
}
} However the search does not find any: ➜ knife search node 'provider:*' -i
0 items found
# versus chef-zero with local data
➜ knife search node 'provider:*' -iz
5 items found
node01.domain.com
node02.domain.com
node03.domain.com
node04.domain.com
node05.domain.com any idea?
Env/config:
Update: Okay, forgot to change the separator:
works |
Looks like the indexation of arrays in attributes into search does not work properly:
The attribute is an array created by a custom ohai plugin that just queries the local dockerd about all images, containers, volumes: "automatic": {
"corp_docker": {
"containers": [
{
...
},
{
...
}
],
"images": [
{
"Containers": -1,
"Created": 1502464017,
"Labels": {
},
"ParentId": "",
"RepoDigests": [
"hub.docker.corp/app1@sha256:...."
],
"RepoTags": [
"hub.docker.corp/app1:unstable"
],
"SharedSize": -1,
"Size": 1004576564,
"VirtualSize": 1004576564,
"id": "sha256:..."
},
{
"Containers": -1,
"Created": 1502464017,
"Labels": {
},
"ParentId": "",
"RepoDigests": [
"hub.docker.corp/app2@sha256:...."
],
"RepoTags": [
"hub.docker.corp/app2:unstable"
],
"SharedSize": -1,
"Size": 1004576564,
"VirtualSize": 1004576564,
"id": "sha256:..."
},
...
}
}
}
We have one case where we use this information to find out all nodes that have a specific image deployed and/or running a container with that image.
Guess I'll have to refactor our plugin to just dump the data in hashes into ohai/chef… |
Thanks for the heads up on that being broken too. I think this is a regression, although from when I'm not sure yet. I do know where it would be coming from though. |
It just occurred to me that that thing with the arrays not being broken down correctly might be the source of the ridiculous memory usage (and why I wasn't able to duplicate it, because I was creating large numbers of fauxhai nodes that IIRC don't have lots of that). I'll run some more tests for that. |
I have some changes brewing in the Thanks again for being such a thorough tester, and thanks for your patience. |
Thanks! I've only 2 days a week time for ops, so I'm sorry my response time is a high. FYI: Here's an example node json of one of our nodes https://gist.github.com/rmoriz/b1e3e335eb95ef268c42502c3c5f4b78 - we are alrady stripping out most ohai data using a custom automatic_attribute_whitelist ["fqdn", "os", "os_version", "ohai_time", "hostname", "keys/ssh", "roles", "recipes", "ipaddress", "ip6address", "platform", "platform_version", "cloud/", "cloud_v2/", "chef_packages", "example-app_docker", "status", "uptime", "uptime_seconds", "node_name", "plattform", "plattform_family", "plattform_version", "os", "os_version", "ohai/"] the |
It needs cleanup and the unicode changes for the parser to be added back in, but the Before it's all merged, I ought to get some actual numbers together for it. |
The https://github.com/ctdk/goiardi/tree/0.11.6-merging branch has the fixes for this with (from informal testing so far) vastly better memory usage, both generally and with the arrays of maps or arrays (and it handles them correctly too), along with the whitelisting for @rmoriz It also has the serf reconnect fix. I just need to run the various chef-pedant tests, and maybe add some more golang tests to test the slices of maps. I'm planning on also getting some better numbers for memory usage. |
Merged and released. |
Just a follow up on this issue which is now solved: I'm still running my personal goiardi instance (~10 nodes) with the gob backend (as a docker container) and it crashed regularly due to low memory (vm has 2GB+1GB SSD swap) while another instance with the pg backend and ~50 nodes works fine for months. I replicated everything locally on my mac using docker and added some debug information to see the amount of various objects in the data store. The number of reports was at 65xxx - just setting the purge option I obviously missed originally didn't help, because purge happens only every 2 hours and the container is then way OOM and restarted. I changed my local instance to purge every minute, waited until the reports were purged and written to disk and then copied After the purge I had to restart goiardi/the container in both cases to regain memory, not sure why. It's running now around 300MB, down from over 2GB. This also solved lots of orphaned temporary index and data files (I assume OOM kill happend before the stream was fully persisted). There is still an unresolved issue with orphaned sandboxes (in my case 127) but I guess they don't take that much of memory. With the pg instance I have a cronjob to cleanup old data via SQL, that part is missing with gob. |
I haven't forgotten about the sandbox thing, I just haven't gotten an answer yet on how long they need to hang around. I've asked again, though, and hopefully I'll get an answer this time. As an aside, I'm also starting to think about re-evaluating the storage backends and whether it's especially worthwhile to keep the in-mem/file backed storage (and the MySQL one, honestly) or tearing out in-mem and totally redoing it, or just settling on Postgres for everything and providing a more convenient docker-compose config to run everything together. The in-mem search, while I'm quite proud of what I did there and has improved a lot thanks to everyone's bug reports, is still kind of a memory hog. I'm certainly open to suggestions on what I should do here. I'm still slowly moving along on the last strange problems for basic Chef Server 12 behavior, although it's constrained by day job stuff and how well I'm feeling on a given day - if I'm worn out and whatever ails me is teeing off on me more than usual, I'm not going to be real productive while riding the train home. |
Good news, @rmoriz. I got an answer finally on the sandbox thing, and they are purged. (Apparently back in the older times it was less often than now or something, but they fixed that up. Certainly when I was digging around in the old 10.x server code it wasn't obvious.) I'll get purging of old sandboxes turned around pretty soon - the in-mem purge may take longer, but the SQL ones won't. Thanks for your patience. |
I've a test setup here running 6 chef-clients against goiardi running on a VPS and goiardi is taking almost the whole memory:
Number of nodes: 6
Options:
'GOIARDI_LOG_EVENTS=1',
'GOIARDI_LOG_EVENT_KEEP=100'
'GOIARDI_LOG_LEVEL=debug'
(stdout shows event purge works)
config file:
Size of
/var/lib/goiardi
is 101MBdocker ps:
Is this expected behaviour? Do you have numbers/experiences on how much memory goiardi needs?
Thanks!
The text was updated successfully, but these errors were encountered: