Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Affine transform from appearance embedding #2222

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Affine transform from appearance embedding #2222

wants to merge 6 commits into from

Conversation

AdamRashid96
Copy link
Contributor

Changes:

  • I added a flag --pipeline.model.use-appearance-embedding that defaults to True
  • When set to false the appearance embedding is not passed into the rgb mlp but is instead used to predict a scale and shift for the bottleneck vector. (Zip-Nerf)

Notes:

  • Results are variable
  • Overall seems to reduce floaters slightly and up the contrast of the scenes
  • Sometimes it can cause some extra noise
  • It also pushed up the ceiling in the Giannini Hall scene

Comparison:
Video
Screen Shot 2023-07-13 at 3 33 15 PM
Screen Shot 2023-07-13 at 3 33 31 PM
Screen Shot 2023-07-13 at 3 34 17 PM
Screen Shot 2023-07-13 at 3 34 32 PM
Screen Shot 2023-07-13 at 3 35 16 PM
Screen Shot 2023-07-13 at 3 35 30 PM

maturk
maturk reviewed Jul 18, 2023
View reviewed changes
nerfstudio/fields/nerfacto_field.py Outdated Show resolved Hide resolved
nerfstudio/models/nerfacto.py Outdated Show resolved Hide resolved
@AdamRashid96 AdamRashid96 requested a review from kerrj August 3, 2023 17:44
tancik
tancik previously approved these changes Aug 17, 2023
View reviewed changes
Copy link
Contributor

@tancik tancik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tancik tancik enabled auto-merge (squash) August 17, 2023 17:56
@tancik tancik disabled auto-merge August 17, 2023 18:00
@kerrj
Copy link
Collaborator

kerrj commented Aug 17, 2023

Right now the behavior of using_appearance embedding is a bit confusing since when it's off, the appearance embedding is still used but it toggles between affine/nonaffine, right? It would be better to have the appearance embedding option expressed as a literal that toggles between 'off', 'affine', and 'mlp' or something

@AdamRashid96
Copy link
Contributor Author

Yeah I agree the wording is confusing and having two different toggles is unnecessary.

@tancik tancik dismissed their stale review August 25, 2023 01:00

Toggles should be added

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@maturk maturk maturk left review comments

@tancik tancik tancik left review comments

@kerrj kerrj Awaiting requested review from kerrj

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@AdamRashid96 @kerrj @tancik @maturk