Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parse samplers on demand #198

Merged
merged 2 commits into from
Sep 12, 2022
Merged

Conversation

HansKristian-Work
Copy link
Collaborator

@HansKristian-Work HansKristian-Work commented Sep 12, 2022

Fixes #196.

Before on the reported archive, there were millions of sampler objects. Memory usage exploded:

Fossilize INFO: =================
Fossilize INFO:  Memory usage:
Fossilize INFO:    #0:   367 MiB resident     3 MiB shared (0 MiB shared metadata).
Fossilize INFO:    #1:  2186 MiB resident   237 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #2:  2156 MiB resident   238 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #3:  2168 MiB resident   238 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #4:  2013 MiB resident   238 MiB shared (242 MiB shared metadata).
Fossilize INFO:  Dirty filesystem writes: 0 MiB.
Fossilize INFO:  IO stall: 0%.
Fossilize INFO:  Num running child processes: 4.
Fossilize INFO: =================

On a pruned archive, there is only 1 surviving sampler, so we get:

Fossilize INFO: =================
Fossilize INFO:  Memory usage:
Fossilize INFO:    #0:   137 MiB resident     3 MiB shared (0 MiB shared metadata).
Fossilize INFO:    #1:   284 MiB resident    89 MiB shared (88 MiB shared metadata).
Fossilize INFO:    #2:   289 MiB resident    89 MiB shared (88 MiB shared metadata).
Fossilize INFO:    #3:   285 MiB resident    89 MiB shared (88 MiB shared metadata).
Fossilize INFO:    #4:   286 MiB resident    89 MiB shared (88 MiB shared metadata).
Fossilize INFO:  Dirty filesystem writes: 9 MiB.
Fossilize INFO:  IO stall: 0%.
Fossilize INFO:  Num running child processes: 4.
Fossilize INFO: =================

This is closer to what we normally expect as a high water mark per process.

With this PR, VkSamplers are only replayed if they are a dependency of a descriptor set layout. Unfortunately, we have a hard time knowing which samplers to record since we don't know at recording time if a sampler is part of a descriptor set layout. We don't want to bloat the runtime memory usage by holding onto an unbound number of samplers "just in case". I'll follow up with another PR to be able to ignore immutable samplers if we know the application (DXVK in this case) does not really make use of them.

Fossilize INFO: =================
Fossilize INFO:  Memory usage:
Fossilize INFO:    #0:   368 MiB resident     3 MiB shared (0 MiB shared metadata).
Fossilize INFO:    #1:   516 MiB resident    90 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #2:   520 MiB resident    89 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #3:   516 MiB resident    89 MiB shared (242 MiB shared metadata).
Fossilize INFO:    #4:   518 MiB resident    89 MiB shared (242 MiB shared metadata).
Fossilize INFO:  Dirty filesystem writes: 18 MiB.
Fossilize INFO:  IO stall: 0%.
Fossilize INFO:  Num running child processes: 4.
Fossilize INFO: =================

This is far more reasonable, but the increased memory usage compared to pruned is the hashmap post-parse needs to holds millions of sampler references. Not recording millions of samplers will fix this as well down the line.

If a game recorded a ton of samplers that were not used in a descriptor
set layout, we risk going OOM.
@HansKristian-Work HansKristian-Work merged commit 063fbea into master Sep 12, 2022
@HansKristian-Work HansKristian-Work deleted the parse-samplers-on-demand branch September 12, 2022 12:34
Copy link

@Frogging101 Frogging101 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

fossilize-replay uses up too much memory (Bioshock Infinite)
2 participants