-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathpaint3D.html
127 lines (123 loc) · 8.59 KB
/
paint3D.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
<!DOCTYPE html>
<html>
<head>
<title>Jason Petersen - Paint 3D Game Objects Using Generative AI</title>
<link rel="stylesheet" type="text/css" href="styles.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css">
</head>
<body>
<header>
<div class="header-content">
<h1>Jason Petersen</h1>
<div class="social-icons">
<a href="https://github.com/jpetersen5" target="_blank"><i class="fab fa-github"></i><span class="username"> jpetersen5</span></a>
<a href="https://linkedin.com/in/jason-t-petersen" target="_blank"><i class="fab fa-linkedin"></i><span class="username"> jason-t-petersen</span></a>
</div>
</div>
<p>University of California, Merced '23 - BS Computer Science & Engineering</p>
<nav>
<a class="nav-button" href="index.html#about">About Me</a>
<a class="nav-button" href="index.html#portfolio">Portfolio</a>
<a class="nav-button" href="index.html#contact">Contact Me</a>
</nav>
</header>
<section id="project-detail">
<h3>Painting 3D Game Objects Using Generative AI with Hypothetic</h3>
<div class="project-introduction">
<p>Over the course of an 8-week
<a href="https://www.parkerdewey.com/career-launchers" target="_blank" class="tooltip">
micro-internship
<span class="tooltiptext">Micro-internships via Parker Dewey are short-term, paid, professional projects that are open to all college students and recent graduates of U.S.-based institutions.<br>https://www.parkerdewey.com/career-launchers</span>
</a>
with
<a href="https://www.hypothetic.art/" target="_blank" class="tooltip">
Hypothetic,
<span class="tooltiptext">Enabling a new generation of 3D creators through the power of generative AI and machine learning.<br>https://www.hypothetic.art/</span>
</a>
facilitated by
<a href="https://www.openavenuesfoundation.org/" target="_blank" class="tooltip">
OpenAvenues,
<span class="tooltiptext">OpenAvenues connects international and U.S. students with industry experts in projects that are designed to built experiences and portfolios.<br>https://www.openavenuesfoundation.org/</span>
</a>
I had the opportunity to work closely with generative AI models. My task was to leverage the capabilities of
<a href="https://huggingface.co/blog/controlnet" target="_blank" class="tooltip">
ControlNet,
<span class="tooltiptext">Adds conditional controlling to text-to-image diffusion models.<br>https://huggingface.co/blog/controlnet</span>
</a>
a cutting-edge generative AI model, to automatically texture 3D objects.
</p>
<p>The primary tool I used was
<a href="https://pytorch3d.org/" target="_blank" class="tooltip">
PyTorch3D,
<span class="tooltiptext">A library for deep learning with 3D data.<br>https://pytorch3d.org/</span>
</a>
a highly versatile library from
<a href="https://ai.meta.com/research/" target="_blank" class="tooltip">
Facebook AI Research
<span class="tooltiptext">Meta research to understand and develop AI systems by advancing the longer-term academic problems surrounding the field.<br>https://ai.meta.com/research/</span>
</a>
(FAIR) that provides a set of utilities and models for 3D computer vision research. PyTorch3D played a crucial role in handling the 3D data and applying the textures generated by ControlNet.
</p>
<p>ControlNet itself is a part of the
<a href="https://huggingface.co/docs/diffusers/index" target="_blank" class="tooltip">
Diffusers library
<span class="tooltiptext">Library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules.<br>https://huggingface.co/docs/diffusers/index</span>
</a>
of generative models that can be found on
<a href="https://huggingface.co/" target="_blank" class="tooltip">
HuggingFace,
<span class="tooltiptext">AI company and community that develops ML tools and hosts thousands of AI models.<br>https://huggingface.co/</span>
</a>
an AI company and community that develops ML tools and hosts thousands of AI models. These Diffusers models are designed to generate high-quality, realistic textures by learning from a large dataset of examples. ControlNet is unique for its ability to use multiple input images and ControlNet models in a single generation pipeline. Additionally, I used
<a href="https://opencv.org/" target="_blank" class="tooltip">
OpenCV,
<span class="tooltiptext">An open source computer vision and machine learning software library built to provide a common infrastructure for computer vision applications and to accelerate the use of machine perception.<br>https://opencv.org/</span>
</a>
a popular library for computer vision tasks, for extra image processing and manipulation, especially in the creation of masks.
</p>
<p>The overall project goal was about understanding and applying these ML and 3D concepts from bleeding-edge research to gain experience and knowledge. In the course of the project, I found it useful to dissect and analyze the
<a href="https://texturepaper.github.io/TEXTurePaper/static/paper.pdf" target="_blank" class="tooltip">
TEXTure paper:
<span class="tooltiptext">Research paper that presents novel concepts for fully texturing 3D objects using AI as well as transfering textures from images.<br>https://texturepaper.github.io/</span>
</a>
research that presented novel concepts for texturing 3D objects using AI, which I adapted and incorporated into my work with ControlNet.
</p>
<p>The final version of the project is documented in a Python Notebook hosted on Google's CoLab
<a href="https://colab.research.google.com/drive/165ohPAEUqfsVShgEVJaYz6X1zz7Iw3Kj?usp=sharing" target="_blank" class="tooltip">
here.
<span class="tooltiptext">The finalized project notebook.<br>https://colab.research.google.com/</span>
</a>
</p>
</div>
<br><br>
<div class="image-container">
<script src="https://cdn.jsdelivr.net/gh/buzzfeed/libgif-js@master/libgif.js"></script>
<img class="gif" rel:animated_src="https://raw.githubusercontent.com/jpetersen5/jpetersen5.github.io/main/src/images/cow.gif" rel:auto_play="0" rel:rubbable="0" alt="Cow Preview">
<div class="scrollbar-container">
<input type="range" min="1" max="60" value="30" class="scrollbar">
</div>
<script>
var gif = new SuperGif({ gif: document.querySelector('.gif') });
gif.load(function() {
var scrollbar = document.querySelector('.scrollbar');
scrollbar.addEventListener('input', function() {
gif.move_to(scrollbar.value - 1); // gif frames are 0-indexed
});
});
</script>
</div>
</section>
<section id="contact">
<h2>Contact Me</h2>
<div class="social-icons">
<a href="mailto:jasonpetersen707@gmail.com" target="_blank"><i class="fas fa-envelope"></i><span class="username"> jasonpetersen707@gmail.com</span></a>
<a href="https://github.com/jpetersen5" target="_blank"><i class="fab fa-github"></i><span class="username"> jpetersen5</span></a>
<a href="https://linkedin.com/in/jason-t-petersen" target="_blank"><i class="fab fa-linkedin"></i><span class="username"> jason-t-petersen</span></a>
</div>
</section>
<footer>
<p>Copyright © 2023 Jason Petersen</p>
</footer>
<script src="source.js"></script>
</body>
</html>