Skip to content
Permalink
Browse files

Renamed person segmentation model to bodyPix (#104)

  • Loading branch information...
oveddan authored and nsthorat committed Nov 18, 2018
1 parent 1a81c39 commit 19f1afc8c1f994320306a1ddb28fbb30cc0992ac
Showing with 164 additions and 154 deletions.
  1. 0 {person-segmentation → body-pix}/.npmignore
  2. +39 −33 {person-segmentation → body-pix}/README.md
  3. BIN person-segmentation/person-segmentation.gif → body-pix/body-pix.gif
  4. BIN {person-segmentation → body-pix}/bokeh.gif
  5. BIN {person-segmentation → body-pix}/colored-parts.gif
  6. 0 {person-segmentation → body-pix}/demos/.babelrc
  7. +10 −10 {person-segmentation → body-pix}/demos/README.md
  8. +2 −2 {person-segmentation → body-pix}/demos/index.html
  9. +7 −10 {person-segmentation → body-pix}/demos/index.js
  10. +1 −1 {person-segmentation → body-pix}/demos/package.json
  11. 0 {person-segmentation → body-pix}/demos/part_color_scales.js
  12. +0 −5 {person-segmentation → body-pix}/demos/yarn.lock
  13. +7 −7 {person-segmentation → body-pix}/package.json
  14. +3 −13 {person-segmentation → body-pix}/rollup.config.js
  15. 0 {person-segmentation → body-pix}/run_tests.ts
  16. BIN {person-segmentation → body-pix}/segmentation.gif
  17. 0 {person-segmentation → body-pix}/src/blur.ts
  18. +22 −18 person-segmentation/src/person_segmentation_model.ts → body-pix/src/body_pix_model.ts
  19. +5 −4 person-segmentation/src/person_segmentation_test.ts → body-pix/src/body_pix_test.ts
  20. 0 {person-segmentation → body-pix}/src/checkpoints.ts
  21. 0 {person-segmentation → body-pix}/src/decode_part_map.ts
  22. +1 −1 {person-segmentation → body-pix}/src/index.ts
  23. 0 {person-segmentation → body-pix}/src/mobilenet.ts
  24. 0 {person-segmentation → body-pix}/src/model_weights.ts
  25. +26 −21 {person-segmentation → body-pix}/src/output_rendering_util.ts
  26. 0 {person-segmentation → body-pix}/src/part_channels.ts
  27. +16 −0 body-pix/src/types.ts
  28. +4 −4 {person-segmentation → body-pix}/src/util.ts
  29. 0 {person-segmentation → body-pix}/tsconfig.json
  30. 0 {person-segmentation → body-pix}/tslint.json
  31. +19 −19 {person-segmentation → body-pix}/yarn.lock
  32. +0 −4 person-segmentation/src/types.ts
  33. +1 −1 posenet/demos/README.md
  34. +1 −1 posenet/package.json
File renamed without changes.
@@ -1,10 +1,10 @@
# Person Segmentation in the Browser
# BodyPix - Person Segmentation in the Browser

This package contains a standalone model called PersonSegmentation, as well as some demos, for running real-time person and body part segmentation in the browser using TensorFlow.js.
This package contains a standalone model called BodyPix, as well as some demos, for running real-time person and body part segmentation in the browser using TensorFlow.js.

[Try the demo here!](https://storage.googleapis.com/tfjs-models/demos/person-segmentation/camera.html)
[Try the demo here!](https://storage.googleapis.com/tfjs-models/demos/body-pix/camera.html)

![Person Segmentation](person-segmentation.gif)
![BodyPix](body-pix.gif)

This model can be used to segment an image into pixels that are and are not part of a person, and into
pixels that belong to each of twenty-four body parts. It works for a single person, and its ideal use case is for when there is only one person centered in an input image or video. It can be combined with a person
@@ -18,35 +18,35 @@ You can use this as standalone es5 bundle like this:

```html
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@0.13.3"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/person-segmentation@0.0.4"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/body-pix@0.0.4"></script>
```

Or you can install it via npm for use in a TypeScript / ES6 project.

```sh
npm install @tensorflow-models/person-segmentation
npm install @tensorflow-models/body-pix
```

## Usage

Either a person or part of the body can be segmented in an image.
Each methodology has similar input parameters with different outputs.

### Loading a pre-trained PersonSegmenation Model
### Loading a pre-trained BodyPix Model

In the first step of segmentation, an image is fed through a pre-trained model. PersonSegmentation **comes with a few different versions of the model,** each corresponding to a MobileNet v1 architecture with a specific multiplier. To get started, a model must be loaded from a checkpoint, with the MobileNet architecture specified by the multiplier:
In the first step of segmentation, an image is fed through a pre-trained model. BodyPix **comes with a few different versions of the model,** each corresponding to a MobileNet v1 architecture with a specific multiplier. To get started, a model must be loaded from a checkpoint, with the MobileNet architecture specified by the multiplier:

```javascript
const net = await personSegmentation.load(multiplier);
const net = await bodyPix.load(multiplier);
```

#### Inputs

* **multiplier** - An optional number with values: `1.0`, `0.75`, or `0.50`, `0.25`. Defaults to `0.75`. It is the float multiplier for the depth (number of channels) for all convolution operations. The value corresponds to a MobileNet architecture and checkpoint. The larger the value, the larger the size of the layers, and more accurate the model at the cost of speed. Set this to a smaller value to increase speed at the cost of accuracy.

**By default,** PersonSegmenation loads a model with a **`0.75`** multiplier. This is recommended for computers with **mid-range/lower-end GPUS.** A model with a **`1.00`** muliplier is recommended for computers with **powerful GPUS.** A model with a **`0.50`** or **`0.25`** architecture is recommended for **mobile.**
**By default,** BodyPix loads a model with a **`0.75`** multiplier. This is recommended for computers with **mid-range/lower-end GPUS.** A model with a **`1.00`** muliplier is recommended for computers with **powerful GPUS.** A model with a **`0.50`** or **`0.25`** architecture is recommended for **mobile.**

### Person Segmentation
### Person segmentation

Person segmentation segments an image into pixels that are and aren't part of a person.
It returns a binary array with 1 for the pixels that are part of the person, and 0 otherwise. The array size corresponds to the number of pixels in the image.
@@ -55,7 +55,7 @@ It returns a binary array with 1 for the pixels that are part of the person, and
![Segmentation](segmentation.gif)

```javascript
const net = await personSegmentation.load();
const net = await bodyPix.load();
const segmentation = await net.estimatePersonSegmentation(image, flipHorizontal, outputStride, segmentationThreshold);
```
@@ -72,7 +72,7 @@ around a person but may result in some pixels being that are part of a person be

#### Returns

A binary array with 1 for the pixels that are part of the person, and 0 otherwise. The array size corresponds to the number of pixels in the image.
An object containing a width, height, and a binary array with 1 for the pixels that are part of the person, and 0 otherwise. The array size corresponds to the number of pixels in the image. The width and height correspond to the dimensions of the image the binary array is shaped to, which are the same dimensions of the input image.

#### Example Usage

@@ -83,8 +83,8 @@ A binary array with 1 for the pixels that are part of the person, and 0 otherwis
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@0.13.3"></script>
<!-- Load PersonSegmentation -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/person-segmentation@0.0.4"></script>
<!-- Load BodyPix -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/body-pix@0.0.4"></script>
</head>

<body>
@@ -98,7 +98,7 @@ A binary array with 1 for the pixels that are part of the person, and 0 otherwis
var imageElement = document.getElementById('cat');
personSegmentation.load().then(function(net){
bodyPix.load().then(function(net){
return net.estimatePersonSegmentation(imageElement, flipHorizontal, outputStride, segmentationThreshold)
}).then(function(segmentation){
console.log(segmentation);
@@ -110,16 +110,16 @@ A binary array with 1 for the pixels that are part of the person, and 0 otherwis
###### via NPM

```javascript
import * as personSegmentation from '@tensorflow-models/person-segmentation';
import * as bodyPix from '@tensorflow-models/body-pix';
const outputStride = 16;
const flipHorizontal = false;
const segmentationThreshold = 0.5;
const imageElement = document.getElementById('cat');
// load the PersonSegmentation model from a checkpoint
const net = await personSegmentation.load();
// load the BodyPix model from a checkpoint
const net = await bodyPix.load();
const segmentation = await net.estimatePersonSegmentation(imageElement, flipHorizontal, outputStride, segmentationThreshold);
@@ -130,12 +130,15 @@ console.log(segmentation);
which would produce the output:

```javascript
Uint8Array(307200) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, …]
]
// an array of 307200 values are returned, one for each pixel of the 640x480 image that was passed to the function.
{
width: 640,
height: 480,
data: Uint8Array(307200) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, …]
}
// an array of 307200 values is returned, one for each pixel of the 640x480 image that was passed to the function.
```

An example of applying a [bokeh effect](https://www.nikonusa.com/en/learn-and-explore/a/tips-and-techniques/bokeh-for-beginners.html) can be seen by running the [demo](https://storage.googleapis.com/tfjs-models/demos/person-segmentation/camera.html):
An example of applying a [bokeh effect](https://www.nikonusa.com/en/learn-and-explore/a/tips-and-techniques/bokeh-for-beginners.html) can be seen by running the [demo](https://storage.googleapis.com/tfjs-models/demos/body-pix/camera.html):


![Bokeh](bokeh.gif)
@@ -149,7 +152,7 @@ It returns an array with a part id from 0-24 for the pixels that are part of a c
![Colored Part Image](colored-parts.gif)

```javascript
const net = await personSegmentation.load();
const net = await bodyPix.load();
const partSegmentation = await net.estimatePartSegmentation(image, flipHorizontal, outputStride, segmentationThreshold);
```
@@ -166,7 +169,7 @@ around a person but may result in some pixels being that are part of a person be

#### Returns

An array with a part id from 0-24 for the pixels that are part of a corresponding body part, and -1 otherwise. The array size corresponds to the number of pixels in the image.
An object containing a width, height, and an array with a part id from 0-24 for the pixels that are part of a corresponding body part, and -1 otherwise. The array size corresponds to the number of pixels in the image. The width and height correspond to the dimensions of the image the array is shaped to, which are the same dimensions of the input image.

#### Example Usage

@@ -177,8 +180,8 @@ An array with a part id from 0-24 for the pixels that are part of a correspondin
<head>
<!-- Load TensorFlow.js -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@0.13.3"></script>
<!-- Load PersonSegmentation -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/person-segmentation@0.0.4"></script>
<!-- Load BodyPix -->
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/body-pix@0.0.4"></script>
</head>

<body>
@@ -192,7 +195,7 @@ An array with a part id from 0-24 for the pixels that are part of a correspondin
var imageElement = document.getElementById('cat');
personSegmentation.load().then(function(net){
bodyPix.load().then(function(net){
return net.estimatePartSegmentation(imageElement, flipHorizontal, outputStride, segmentationThreshold)
}).then(function(partSegmentation){
console.log(partSegmentation);
@@ -204,7 +207,7 @@ An array with a part id from 0-24 for the pixels that are part of a correspondin
###### via NPM

```javascript
import * as personSegmentation from '@tensorflow-models/person-segmentation';
import * as bodyPix from '@tensorflow-models/body-pix';
const outputStride = 16;
const flipHorizontal = false;
@@ -213,7 +216,7 @@ const segmentationThreshold = 0.5;
const imageElement = document.getElementById('cat');
// load the person segmentation model from a checkpoint
const net = await personSegmentation.load();
const net = await bodyPix.load();
const segmentation = await net.estimatePartSegmentation(imageElement, flipHorizontal, outputStride, segmentationThreshold);
@@ -224,9 +227,12 @@ console.log(segmentation);
which would produce the output:

```javascript
Float32Array(307200) [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 3, 3, 3, 3, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 0, 0, 0, 0, 1, 1, 2, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 15, 15, 15, 15, 16, 16, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 23, 23, 23, 22, 22, -1, -1, -1, -1,  
…]
// an array of 307200 values are returned, one for each pixel of the 640x480 image that was passed to the function.
{
width: 680,
height: 480,
data: Int32Array(307200) [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 3, 3, 3, 3, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 0, 0, 0, 0, 0, 1, 1, 2, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 15, 15, 15, 15, 16, 16, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, 23, 23, 23, 22, 22, -1, -1, -1, -1,  …]
}
// an array of 307200 values is returned, one for each pixel of the 640x480 image that was passed to the function.
```

## Developing the Demos
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
@@ -1,4 +1,4 @@
# person-segmentation Demos
# body-pix Demos

## Contents

@@ -9,7 +9,7 @@ The demo shows how to estimate segmentation in real-time from a webcam video str
cd into the demos folder:

```sh
cd person-segmentation/demos
cd body-pix/demos
```

Install dependencies and prepare the build directory:
@@ -24,24 +24,24 @@ To watch files for changes, and launch a dev server:
yarn watch
```

## If you are developing person-segmentation locally, and want to test the changes in the demos
## If you are developing body-pix locally, and want to test the changes in the demos

Install yalc:
```sh
npm i -g yalc
```

cd into the person-segmentation folder:
cd into the body-pix folder:
```sh
cd person-segmentation
cd body-pix
```

Install dependencies:
```sh
yarn
```

Publish person-segmentation locally:
Publish body-pix locally:
```sh
yalc push
```
@@ -53,19 +53,19 @@ cd demos
yarn
```

Link the local person-segmentation to the demos:
Link the local body-pix to the demos:
```sh
yalc link \@tensorflow-models/person-segmentation
yalc link @tensorflow-models/body-pix
```

Start the dev demo server:
```sh
yarn watch
```

To get future updates from the person-segmentation source code:
To get future updates from the body-pix source code:
```
# cd up into the person-segmentation directory
# cd up into the body-pix directory
cd ../
yarn build && yalc push
```
@@ -2,7 +2,7 @@
<html>

<head>
<title>Person Segmentation - With a Webcam Demo</title>
<title>BodyPix - With a Webcam Demo</title>
<style>
.footer {
position: fixed;
@@ -47,7 +47,7 @@
<div class="footer">
<div class="footer-text">
<p>
The Person Segmentation model can estimate which pixels in an image are part of a person, and which pixels
The BodyPix model can estimate which pixels in an image are part of a person, and which pixels
are part of each of 24 body parts. It works on a single person, and such <strong>works best</strong> when <strong>one person is present</strong> in an image.
<br>
<br> The <strong>output stride</strong> and <strong>model (indicated by mobileNetArchitecture)</strong> have the largest effects on accuracy/speed. A <i>higher</i> output stride results in lower accuracy but higher speed. A <i>larger</i> model, indicated by the <i>mobileNetArchitecture</i> dropdown, results in higher accuracy but lower speed.
@@ -14,7 +14,7 @@
* limitations under the License.
* =============================================================================
*/
import * as personSegmentation from '@tensorflow-models/person-segmentation';
import * as bodyPix from '@tensorflow-models/body-pix';
import dat from 'dat.gui';
import Stats from 'stats.js';

@@ -208,8 +208,7 @@ function segmentBodyInRealTime(video, net) {

// Load the PoseNet model weights for either the 0.50, 0.75, 1.00, or 1.01
// version
guiState.net =
await personSegmentation.load(+guiState.changeToArchitecture);
guiState.net = await bodyPix.load(+guiState.changeToArchitecture);

guiState.changeToArchitecture = null;
}
@@ -229,12 +228,11 @@ function segmentBodyInRealTime(video, net) {

switch (guiState.segmentation.effect) {
case 'mask':
personSegmentation.drawBodyMaskOnCanvas(
video, bodySegmentation, canvas);
bodyPix.drawBodyMaskOnCanvas(video, bodySegmentation, canvas);

break;
case 'bokeh':
personSegmentation.drawBokehEffectOnCanvas(
bodyPix.drawBokehEffectOnCanvas(
canvas, video, bodySegmentation,
+guiState.segmentation.bokehBlurAmount);
break;
@@ -245,7 +243,7 @@ function segmentBodyInRealTime(video, net) {
video, flipHorizontal, outputStride,
guiState.segmentation.segmentationThreshold);

personSegmentation.drawBodySegmentsOnCanvas(
bodyPix.drawBodySegmentsOnCanvas(
canvas, video, partSegmentation,
partColorScales[guiState.partMap.colorScale]);

@@ -268,9 +266,8 @@ function segmentBodyInRealTime(video, net) {
* available camera devices, and setting off the detectPoseInRealTime function.
*/
export async function bindPage() {
// Load the PersonSegmentation model weights with architecture 0.75
const net =
await personSegmentation.load(+guiState.input.mobileNetArchitecture);
// Load the BodyPix model weights with architecture 0.75
const net = await bodyPix.load(+guiState.input.mobileNetArchitecture);

document.getElementById('loading').style.display = 'none';
document.getElementById('main').style.display = 'block';
@@ -9,7 +9,7 @@
"node": ">=8.9.0"
},
"dependencies": {
"@tensorflow-models/person-segmentation": "0.0.4",
"@tensorflow-models/body-pix": "0.0.1",
"@tensorflow/tfjs": "0.13.3",
"stats.js": "0.17.0"
},
File renamed without changes.
@@ -695,11 +695,6 @@
resolved "https://registry.yarnpkg.com/@protobufjs/utf8/-/utf8-1.1.0.tgz#a777360b5b39a1a2e5106f8e858f2fd2d060c570"
integrity sha1-p3c2C1s5oaLlEG+OhY8v0tBgxXA=

"@tensorflow-models/person-segmentation@0.0.4":
version "0.0.4"
resolved "https://registry.yarnpkg.com/@tensorflow-models/person-segmentation/-/person-segmentation-0.0.4.tgz#f8c7726051007266f59db0c4163ebd50a7b91bd5"
integrity sha512-LBBUi9DCj1um3iHd9ZXJWmqsHwBdZVh4Ixm5TFw0Bt8xXN0t2kPyq2/6W1n+j+EKhGwm68qBcoWKDZiDZ/eckQ==

"@tensorflow/tfjs-converter@0.6.5":
version "0.6.5"
resolved "https://registry.yarnpkg.com/@tensorflow/tfjs-converter/-/tfjs-converter-0.6.5.tgz#71d28889e4236e8178d8cb849e741751011cb1f7"
@@ -1,12 +1,12 @@
{
"name": "@tensorflow-models/person-segmentation",
"version": "0.0.4",
"description": "Pretrained Person Segmentation model in tensorflow.js",
"name": "@tensorflow-models/body-pix",
"version": "0.0.1",
"description": "Pretrained BodyPix model in TensorFlow.js",
"main": "dist/index.js",
"jsnext:main": "dist/person-segmentation.esm.js",
"module": "dist/person-segmentation.esm.js",
"unpkg": "dist/person-segmentation.min.js",
"jsdelivr": "dist/person-segmentation.min.js",
"jsnext:main": "dist/body-pix.esm.js",
"module": "dist/body-pix.esm.js",
"unpkg": "dist/body-pix.min.js",
"jsdelivr": "dist/body-pix.min.js",
"types": "dist/index.d.ts",
"repository": {
"type": "git",
Oops, something went wrong.

0 comments on commit 19f1afc

Please sign in to comment.
You can’t perform that action at this time.