Skip to content

Conversation

liiir1985
Copy link
Contributor

@liiir1985 liiir1985 commented May 14, 2020

List.RemoveAt method would copy the elements around, doing so in a huge enumeration breaks the performance,

Purpose of this PR

By stripping down a large amount of shader variants the original implementation would fail to finish the enumeration in a reasonable time, RemoveAt method will copy all elements behind the removed element in every iteration which kills the performance. This PR fixes this issue by adding the valid variants into a temporary list and copy it to the final list with one additional loop

Testing status

Manual Tests

I've tested the implementation in a project with 850,000 unstriped shader variants, the process time reduced from over 2 hours to within 2 seconds

liiir1985 added 2 commits May 14, 2020 10:51
List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance
@liiir1985 liiir1985 marked this pull request as ready for review May 14, 2020 11:08
@liiir1985 liiir1985 requested a review from a team as a code owner May 14, 2020 11:08
@liiir1985 liiir1985 requested a review from martint-unity May 14, 2020 11:08
Copy link
Contributor

@phi-lira phi-lira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this and submitting the PR. I've made a suggestion to check performance of another solution.

}
compilerDataList.Clear();
foreach(var i in final)
compilerDataList.Add(i);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried about this solution, the list here might still be huge. I've looked at HDRP solution.
They do it in place with a single list. Instead of removing, they swap i with last item. Then they decrement last item index. At the end, they remove all elements at once from last item index to list size. Would you be interested in looking at this solution and comparing performance?

// here's the HDRP code, ignore the inner loop about asset
https://github.com/Unity-Technologies/Graphics/blob/master/com.unity.render-pipelines.high-definition/Editor/BuildProcessors/HDRPPreprocessShaders.cs#L256-L295

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the size of the list is the final result, I think it can't be reduced further more? it only adds the variants which are not striped, but sure I'd like to take a look at the implementation of HDRP

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The implementation of HDRP is indeed better, so I modified the PR to adopt HDRP's way for doing it

@phi-lira phi-lira changed the base branch from master to universal/improve-shader-stripping May 15, 2020 09:02
@phi-lira
Copy link
Contributor

phi-lira commented May 15, 2020

Changed base to a local branch so we run test automation. Thanks for adding changelog!

@phi-lira phi-lira merged commit 750e587 into Unity-Technologies:universal/improve-shader-stripping May 15, 2020
phi-lira added a commit that referenced this pull request May 26, 2020
* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
phi-lira added a commit that referenced this pull request May 27, 2020
* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
phi-lira added a commit that referenced this pull request May 27, 2020
* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
# Conflicts:
#	com.unity.render-pipelines.universal/CHANGELOG.md
phi-lira added a commit that referenced this pull request May 27, 2020
* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
phi-lira added a commit that referenced this pull request May 27, 2020
* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
# Conflicts:
#	com.unity.render-pipelines.universal/CHANGELOG.md

# Conflicts:
#	com.unity.render-pipelines.universal/CHANGELOG.md
Kleptine pushed a commit to Pontoco/Graphics that referenced this pull request Jul 16, 2020
…) (Unity-Technologies#500)

* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
# Conflicts:
#	com.unity.render-pipelines.universal/CHANGELOG.md
#	com.unity.render-pipelines.universal/Editor/ShaderPreprocessor.cs
Kleptine pushed a commit to Pontoco/Graphics that referenced this pull request Jul 16, 2020
…) (Unity-Technologies#500)

* Fixed performance problem of ShaderPreprocessor

List.RemoveAt method would copy the elements around, doing so in a huge iteration breaks the performance

* Added changlog for shader preprocessor fix

* Improved the variant striping

* Fixed typo

Co-authored-by: liiir1985 <liiir1985@users.noreply.github.com>
# Conflicts:
#	com.unity.render-pipelines.universal/CHANGELOG.md
#	com.unity.render-pipelines.universal/Editor/ShaderPreprocessor.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants