Skip to content

Updater: target hash calculation should not read the whole file in memory #1215

@jku

Description

@jku

Description of issue or feature request:

This is not a bug (although it could be on memory limited client device), but a performance improvement:

After PR #1202 there is only one place where the updater loads the whole target file in memory. We should avoid doing that as targets could be very large and memory could be limited.

Current behavior:

_check_hashes() does this:

digest_object = securesystemslib.hash.digest(algorithm)
digest_object.update(file_object.read())
computed_hash = digest_object.hexdigest()

Expected behavior:
something like handwaves

digest_object = securesystemslib.hash.digest(algorithm)
while True:
    chunk = file_object.read(CHUNK_SIZE)
    if not chunk:
        break
    digest_object.update(chunk)
computed_hash = digest_object.hexdigest()

or even more simply just let SSLib handle this with its default chunk size:

digest_object = securesystemslib.hash.digest_fileobject(file_object, algorithm)
computed_hash = digest_object.hexdigest()

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions