[clang-doc] JSONGenerator creates bad index.json files

[ilovepi]

when using the mustache backend, I'm seeing very odd index.html files. These seem correct from the corresponding JSON, so it's a problem w/in the JSONGenerator.

What seems to be happening is that there is some kind of race to write the index.json file, and things that used to be written to a local index.json in the directory hierarchy are now racing w/ one another in the flattened structure, since they're all just index.json.

I think we either need to go back to using a proper directory structure, or we need an alternate way to deal with this in the flattened structure.

One option is to use the path in the filename. another is to use some kind of UUID and have a json file with the mapping that can be used, either from the JS or to generate the right links in the template. I guess we could also use something like we do for too long file names to make them unique.

@evelez7 @petrhosek do you have any thoughts?

[evelez7]

This sounds like a naming issue to me. index.json should correspond to the global namespace. When it's navigated to, it should display all of its children e.g. "nested" namespaces (which is just the first declaration of a namespace like llvm), classes, etc. So an index.json file for the namespace llvm should just be called llvm.json, then llvm::clang should be called clang.json. Those filenames might not be resolved correctly.

If the generator is encountering global namespace entities out of order while iterating through Infos, then could there be problems as it writes more stuff to index.json?

[petrhosek]

This sounds like a naming issue to me. index.json should correspond to the global namespace. When it's navigated to, it should display all of its children e.g. "nested" namespaces (which is just the first declaration of a namespace like llvm), classes, etc. So an index.json file for the namespace llvm should just be called llvm.json, then llvm::clang should be called clang.json. Those filenames might not be resolved correctly.

Do we use only the innermost namespace name as the filename? What happens if we have namespaces like llvm::tools and llvm::clang::tools?

[evelez7]

Do we use only the innermost namespace name as the filename? What happens if we have namespaces like llvm::tools and llvm::clang::tools?

I was thinking of this while I wrote that and I'm not sure. I need to check this as it might present another problem. We'll probably have to write out the full path and default to our hybrid USR approach. So something like llvm_tools.json, then concatenate the USR if it's too long.

[evelez7]

I think I found the cause of this.

llvm-project/clang-tools-extra/clang-doc/JSONGenerator.cpp

Lines 585 to 589 in 78b80a1

	} else if (I->IT == InfoType::IT_namespace && I->Name != "")
	// Serialize the global namespace as index.json
	FileName = I->Name;
	else
	FileName = I->getFileBaseName();

In determineFileName, a namespace Info with an empty name is the global namespace hence it's index.json. A namespace Info without an empty name calls Info::getFileBaseName.

llvm-project/clang-tools-extra/clang-doc/Representation.cpp

Lines 233 to 236 in 78b80a1

	llvm::SmallString<16> Info::getFileBaseName() const {
	if (IT == InfoType::IT_namespace)
	return llvm::SmallString<16>("index");

Which just returns "index" for any namespace.

[ilovepi]

Which just returns "index" for any namespace.

the empty namespace == the global namespace, so they all just go there, and if they don't all "append" (for lack of a better term) to that file, then I think we lose whatever is there.

[evelez7]

It turns out that snippet of code is correct, but not the best so I might change it anyways. I think this might have to do with the way we open files, which is not in append mode.

raw_fd_ostream InfoOS(Group.getKey(), FileErr, sys::fs::OF_Text);

So if there are multiple Infos for one path, then not everything is being written correctly.