New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable configuration of local store size limits #11777
Merged
Merged
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -251,11 +251,59 @@ def from_options( | |
) | ||
|
||
|
||
@dataclass(frozen=True) | ||
class LocalStoreOptions: | ||
"""A collection of all options related to the local store. | ||
|
||
TODO: These options should move to a Subsystem once we add support for "bootstrap" Subsystems (ie, | ||
allowing Subsystems to be consumed before the Scheduler has been created). | ||
""" | ||
|
||
store_dir: str | ||
processes_max_size_bytes: int | ||
files_max_size_bytes: int | ||
directories_max_size_bytes: int | ||
|
||
def target_total_size_bytes(self) -> int: | ||
"""Returns the target total size of all of the stores. | ||
|
||
The `max_size` values are caps on the total size of each store: the "target" size | ||
is the size that garbage collection will attempt to shrink the stores to each time | ||
it runs. | ||
|
||
NB: This value is not currently configurable, but that could be desirable in the future. | ||
""" | ||
max_total_size_bytes = ( | ||
self.processes_max_size_bytes | ||
+ self.files_max_size_bytes | ||
+ self.directories_max_size_bytes | ||
) | ||
return max_total_size_bytes // 10 | ||
|
||
@classmethod | ||
def from_options(cls, options: OptionValueContainer) -> LocalStoreOptions: | ||
return cls( | ||
store_dir=Path(options.local_store_dir).resolve().as_posix(), | ||
stuhood marked this conversation as resolved.
Show resolved
Hide resolved
|
||
processes_max_size_bytes=options.local_store_processes_max_size_bytes, | ||
files_max_size_bytes=options.local_store_files_max_size_bytes, | ||
directories_max_size_bytes=options.local_store_directories_max_size_bytes, | ||
) | ||
|
||
|
||
_CPU_COUNT = ( | ||
len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count() | ||
) or 2 | ||
|
||
|
||
# The time that leases are acquired for in the local store. Configured on the Python side | ||
# in order to ease interaction with the StoreGCService, which needs to be aware of its value. | ||
LOCAL_STORE_LEASE_TIME_SECS = 2 * 60 * 60 | ||
|
||
|
||
MEGABYTES = 1000000 | ||
stuhood marked this conversation as resolved.
Show resolved
Hide resolved
|
||
GIGABYTES = 1000 * MEGABYTES | ||
|
||
|
||
DEFAULT_EXECUTION_OPTIONS = ExecutionOptions( | ||
# Remote execution strategy. | ||
remote_execution=False, | ||
|
@@ -285,6 +333,13 @@ def from_options( | |
remote_execution_overall_deadline_secs=60 * 60, # one hour | ||
) | ||
|
||
DEFAULT_LOCAL_STORE_OPTIONS = LocalStoreOptions( | ||
store_dir=os.path.join(get_pants_cachedir(), "lmdb_store"), | ||
processes_max_size_bytes=(4 * GIGABYTES), | ||
files_max_size_bytes=(128 * GIGABYTES), | ||
directories_max_size_bytes=(4 * GIGABYTES), | ||
stuhood marked this conversation as resolved.
Show resolved
Hide resolved
|
||
) | ||
|
||
|
||
class GlobalOptions(Subsystem): | ||
options_scope = GLOBAL_SCOPE | ||
|
@@ -701,12 +756,13 @@ def register_bootstrap_options(cls, register): | |
), | ||
) | ||
|
||
local_store_dir_flag = "--local-store-dir" | ||
cache_instructions = ( | ||
"The path may be absolute or relative. If the directory is within the build root, be " | ||
"sure to include it in `--pants-ignore`." | ||
) | ||
register( | ||
"--local-store-dir", | ||
local_store_dir_flag, | ||
advanced=True, | ||
help=( | ||
f"Directory to use for the local file store, which stores the results of " | ||
|
@@ -715,7 +771,46 @@ def register_bootstrap_options(cls, register): | |
# This default is also hard-coded into the engine's rust code in | ||
# fs::Store::default_path so that tools using a Store outside of pants | ||
# are likely to be able to use the same storage location. | ||
default=os.path.join(get_pants_cachedir(), "lmdb_store"), | ||
default=DEFAULT_LOCAL_STORE_OPTIONS.store_dir, | ||
) | ||
register( | ||
"--local-store-processes-max-size-bytes", | ||
type=int, | ||
advanced=True, | ||
help=( | ||
"The maximum size in bytes of the local store containing process cache entries. " | ||
f"Stored below `{local_store_dir_flag}`." | ||
), | ||
default=DEFAULT_LOCAL_STORE_OPTIONS.processes_max_size_bytes, | ||
) | ||
register( | ||
"--local-store-files-max-size-bytes", | ||
type=int, | ||
advanced=True, | ||
help=( | ||
"The maximum size in bytes of the local store containing files. " | ||
f"Stored below `{local_store_dir_flag}`." | ||
"\n\n" | ||
"NB: This size value bounds the total size of all files, but (due to sharding of the " | ||
"store on disk) it also bounds the per-file size to (VALUE / 16)." | ||
stuhood marked this conversation as resolved.
Show resolved
Hide resolved
|
||
"\n\n" | ||
"This value doesn't reflect space allocated on disk, or RAM allocated (it " | ||
"may be reflected in VIRT but not RSS). However, the default is lower than you " | ||
"might otherwise choose because macOS creates core dumps that include MMAP'd " | ||
"pages, and setting this too high might cause core dumps to use an unreasonable " | ||
"amount of disk if they are enabled." | ||
), | ||
default=DEFAULT_LOCAL_STORE_OPTIONS.files_max_size_bytes, | ||
) | ||
register( | ||
"--local-store-directories-max-size-bytes", | ||
type=int, | ||
advanced=True, | ||
help=( | ||
"The maximum size in bytes of the local store containing directories. " | ||
f"Stored below `{local_store_dir_flag}`." | ||
), | ||
default=DEFAULT_LOCAL_STORE_OPTIONS.directories_max_size_bytes, | ||
) | ||
Comment on lines
+771
to
809
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. One potential alternative to exposing all three options would be to expose one option, and to then bake in a ratio. |
||
register( | ||
"--named-caches-dir", | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The fact that this is calculated from the new higher default ended up changing the concrete target to a higher value.