Skip to content

Add log connectivity parameter to fossilize-list - #284

Merged
HansKristian-Work merged 36 commits into
ValveSoftware:masterfrom
JohnMega:Add-log-connectivity-parameter-to-fossilize-list2
Oct 9, 2025
Merged

Add log connectivity parameter to fossilize-list#284
HansKristian-Work merged 36 commits into
ValveSoftware:masterfrom
JohnMega:Add-log-connectivity-parameter-to-fossilize-list2

Conversation

@JohnMega

@JohnMega JohnMega commented Oct 2, 2025

Copy link
Copy Markdown
Contributor

This PR adds the --connectivity parameter to fossilize-list. Connectivity in this case refers to the hashes of objects referenced by the hash of the selected tag (via the --tag parameter), or in other words, the hashes of objects associated with that hash.

Example for Graphics Pipeline:
image

@JohnMega

This comment was marked as spam.

Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
bool replayer_create_info_fill(ResourceTag selected_tag, StateReplayer& replayer, ListReplayer& list_replayer, const std::unique_ptr<DatabaseInterface>& input_db)
{
vector<uint8_t> state_db;
for (auto tag : playback_order)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like O(n^2) explosion, which won't fly for larger archives. The outer loop loops over all hashes, and this also loops over all hashes. Any particular reason why it has to be this way?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is done because the replayer needs to know in advance about some Vulcan objects for the selected object, in order to write them to create info. However, I can go through all the database hashes once and collect all the information in an array of structures in the ListReplayer itself. This should be cheaper than what is currently happening.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@HansKristian-Work HansKristian-Work left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design here feels like it would be far simpler if you just replay the blobs with a trivial impl of object = hash, then look at the resulting create infos, and print the relevant objects embedded. Like for pipelines, loop over the stages, and print the module, as well as renderPass/layout, etc.

This implementation seems overly complicated and slow and likely needs a rewrite.

Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
unordered_set<Hash> saved_raytracing_pipelines;
};

map<Hash, SavedHashes> saved_hashes_map;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this an ordered map?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, you need different hash maps per resource type. Hashes from difference resource types are allowed to collide.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated
{
struct SavedHashes
{
unordered_set<Hash> saved_samplers;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like it should just be a vector of pair<tag, hash>. There's no reason why every CLI invocation should give unstable results in stdout due to iterating over these hashmaps. Hashmap is also overkill for this scenario.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated

bool replayer_create_info_fill(ResourceTag selected_tag, StateReplayer& replayer, ListReplayer& list_replayer, const std::unique_ptr<DatabaseInterface>& input_db)
{
// fill Vulcan object data in replayer

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's Vulkan.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated
using namespace Fossilize;
using namespace std;

#define PRINT_SAVED_BUFFER(SavedBufferName, VkObjectName, Tag)\

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need a macro just to print out an array.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated
if (log_connectivity)
{
size_t state_db_size;
if (!input_db->read_entry(tag, hash, &state_db_size, nullptr, 0))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little confused why you're reading the blob here. It's not used for anything?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forgot to clean up after previous commits.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed

Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
{
for (auto par : saved_hashes_map->second)
{
printf(tag_names[par.first - 1 >= tag_names_size ? tag_names_size - 1 : par.first - 1]);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the minus 1?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last index of tag_names

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, why is this not just tag_names[par.first] ? There should be no OOB to worry about and the table could be 1:1 with resource tag types.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason is that RESOURCE_RAYTRACING_PIPELINE = 9

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't the table just accomodate all possibilities, padding with empty strings as necessary? There's no need to complicate this more than it needs to be.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
JohnMega and others added 2 commits October 8, 2025 12:35
Co-authored-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Co-authored-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Comment thread cli/fossilize_list.cpp Outdated
Co-authored-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Co-authored-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Comment thread cli/fossilize_list.cpp Outdated
Co-authored-by: Hans-Kristian Arntzen <post@arntzen-software.no>
Comment thread cli/fossilize_list.cpp Outdated
if (selected_tag != RESOURCE_COMPUTE_PIPELINE)
return true;

auto saved_hash_iter = saved_hashes_map.insert({ hash, saved_hashes_type() }).first;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's much easier to read the plain:

auto &dependencies = saved_hashes_map[hash];
dependencies.push_back(...);

No need to use iterators here.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Comment thread cli/fossilize_list.cpp Outdated
#include "layer/utils.hpp"
#include <memory>
#include <vector>
#include <unordered_set>

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unordered_map?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@HansKristian-Work

Copy link
Copy Markdown
Collaborator

A general tip: When applying code suggestions on github you can batch them up and commit them in one go instead of generating 10 commits for each individual suggestion. (I'll squash it anyway, but reduces clutter a bit).

Vlad added 3 commits October 8, 2025 13:11
…o-fossilize-list2' into Add-log-connectivity-parameter-to-fossilize-list2
Comment thread cli/fossilize_list.cpp Outdated
if (selected_tag != RESOURCE_PIPELINE_LAYOUT)
return true;

saved_hashes_map.insert({ hash, saved_hashes_type() });

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to insert. It's implied by saved_hashes_map[hash].

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, didn't know about it, thanks.

@JohnMega

JohnMega commented Oct 8, 2025

Copy link
Copy Markdown
Contributor Author

I removed the constants in the arguments of some functions (input_db), as the methods called by these constants are not constant at all.

Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
Comment thread cli/fossilize_list.cpp Outdated
@HansKristian-Work
HansKristian-Work merged commit bdaff23 into ValveSoftware:master Oct 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants