Retina encoder: biological encoder for vision WIP #691

breznak · 2019-09-26T14:37:48Z

eye.py is a biological implementation of retina (encoder).
WIP

For #682

eye.py is a biological implementation of retina (encoder). WIP

breznak · 2019-09-26T14:38:56Z

FYI @ctrl-z-9000-times if you have some insights, please share, I'll slowly try to work this

breznak

Steps:

py/htm/encoders/eye.py

ctrl-z-9000-times · 2019-09-27T14:45:06Z

Thank you Breznak for getting the ball rolling on this PR.

I did some basic cleanup. You should now be able to run this locally (without installing my old research repo).

The encoder is now run-able from the command line. It requires a single argument: the image file-path or directory.
$ python3 py/htm/encoders/eye.py ~/Pictures/

some imports for Eye/Retina encoder needed fixing for newer versions

make it a const variable

keep it only as member variable

which retinal pathway to simulate

should be 1/3rd, not 3rd-root

let's you disable/enable plotting, useful for headless mode

self.image is used for both string/data functionality

This reverts commit f259ec6.

breznak

Please have a look at the progress, main changes are

removed Sampler code

Open issues:

how to integrate n saccadic steps into the final image SDR? (logical AND?)
- and related, output SDR seems too dense

bindings/py/packaging/setup.py

py/htm/encoders/eye.py

breznak · 2019-10-02T14:40:18Z

validate the encode on MNIST classification task

I'm running into an import problem, from htm.encoders.eye import Eye, which is strange as GridCellEncoder works fine.

breznak · 2019-10-22T22:15:40Z

Overall this looks good. I think it's close to good enough for this PR.

thank you for reviewing! Addressed some of your concerns, I have more cleanups in another PR, but I broke sth there..so it's probably a good idea to get this into a shippable shape and resolve, and then do followups.

breznak · 2019-10-22T22:41:40Z

TODOs:

I don't know why the test on Linux fails in the CI https://github.com/htm-community/htm.core/pull/691/checks?check_run_id=270681996#step:11:169 (not really descriptive), passes locally.
resolve logpol without cropping to a ROI? Retina encoder: biological encoder for vision WIP #691
import non-breaking improvements from Retina encoder: biological encoder for vision WIP #691
test on MNIST
complex unit tests for features of Retina (lighting, motion).

breznak · 2019-10-23T08:31:24Z

logpolar transform vs. retinal log sampling:
opencv/opencv_contrib#2305

both achieve "highlight in fovea, background less important"
logpolar has the property that rotation,scale are represented as translations, thus have similar representations in encoder (overlap), but what about translation?
retinal log sampling is biologically plausible by the model.
both suffer that we (have to?) crop the image to ROI, and apply the transform there (with fovea in center of image/ROI), this cuts off the info outside of ROI, which would have been reduced anyway but continuously

Tl;DR: can we use Retina.useRetinalLogSampling instead of manual logpolar transform?

for compatibility with encoders

ctrl-z-9000-times · 2019-10-23T15:25:53Z

logpolar has the property that rotation,scale are represented as translations, thus have similar representations in encoder (overlap), but what about translation?

Yes. Assuming the motion is small, the eye's output should still have semantic similarity.

both suffer that we (have to?) crop the image to ROI, and apply the transform there (with fovea in center of image/ROI), this cuts off the info outside of ROI, which would have been reduced anyway but continuously.

No. The area outside of the ROI is lost, not reduced. The things outside of the ROI are outside of the eye's field of view. The peripheral vision needs to be included inside of the ROI.

breznak · 2019-10-23T16:11:30Z

The things outside of the ROI are outside of the eye's field of view. The peripheral vision needs to be included inside of the ROI.

ok, there might be some micommunication of terms on my side, but imagine this case:
"right now I'm reading on the notebook screen with the room in the background":

this is my field of view (FOV, about 160deg horizontally), that could be expressed as a photo taken from my position
my focus (fovea) is on the screen I read text from, which is only a tiny fration of the area on the picture
Q: ROI==FOV vs. ROI=="fovea"(screen)?

This should illustrate that even for ROI (as implemented, the area of image that gets processed, other gets lost), if:

ROI is FOV: we need to be able to specify position, diemeter of retinal fovea.
ROI is "fovea", I think this is a bad design,as it leads to either: too much high-details (whole room in fovea), or almost none peripheral vision (only the "screen" and corners that fit into its bounding box), in reality I am able to notice motion in a large area.

I think to summarize,

The peripheral vision needs to be included inside of the ROI.

if this is true, we need to be able to specify the ratio of fovea/peripheral better

ctrl-z-9000-times · 2019-10-23T16:40:52Z

The ROI is the entire field of view.
The fovea is a small area at the center of the ROI.

we need to be able to specify the ratio of fovea/peripheral better

This is one of the many tuning parameters, IIRC self.fovea_scale

`:

in helper visualization

breznak

Please test this out,
esp. look at:

scaling, how to make it work
if custom log-polar can be replaced by Retina's

breznak · 2019-10-23T18:14:17Z

py/htm/examples/mnist.py

@@ -115,7 +120,10 @@ def main(parameters=default_parameters, argv=None, verbose=True):
    # Training Loop
    for i in range(len(train_images)):
        img, lbl = random.choice(training_data)
-        encode(img, enc)
+        encoder.new_image(img)
+        (enc, _) = encoder.compute()


WIP on MNIST, not yet tuned. I should revert these changes for now.

breznak · 2019-10-23T18:17:55Z

py/htm/encoders/eye.py

+        self.retina_diameter   = int(self.resolution_factor * output_diameter)
+        # Argument fovea_scale  ... proportion of the image (ROI) which will be covered (seen) by
+        # high-res fovea (parvo pathway)
+        self.fovea_scale       = 0.177


I have previously misinterpreted this and self.scale. Rename and change this to fovea_diameter to be clearer?

breznak · 2019-10-23T18:19:03Z

py/htm/encoders/eye.py

+            inputSize            = (self.retina_diameter, self.retina_diameter),
+            colorMode            = color,
+            colorSamplingMethod  = cv2.bioinspired.RETINA_COLOR_BAYER,
+            useRetinaLogSampling = True,)


@ctrl-z-9000-times please compare with this on/off. Can it replace our manual log-polar transformation?

breznak · 2019-10-23T18:20:59Z

py/htm/encoders/eye.py

+        roi.resize( (self.retina_diameter, self.retina_diameter, 3))
+
+        # Mask out areas the eye can't see by drawing a circle boarder.
+        # this represents the "shape" of the sensor/eye (comment out to leave rectangural)


ok to crop to circular region here (and not only in the visualization)? Makes encoder see only ROI as the inner circle.

breznak · 2019-10-23T18:21:39Z

py/htm/encoders/eye.py

+
+        # apply field of view (FOV), rotation
+        self.roi = self._crop_roi()
+        self.roi = self.rotate_(self.roi, self.orientation)


apply rotation to the image itself, instead of separately to output, visualizations, etc

py/htm/encoders/eye.py

where plot was broken with frational scaling. Using cv2.resize() rather than numpy's roi.resize() fixes the issue (numerical problems)

Retina encoder: initial commit

e873102

eye.py is a biological implementation of retina (encoder). WIP

breznak added in_progress encoder research new functionality of HTM theory, research idea labels Sep 26, 2019

breznak assigned breznak and ctrl-z-9000-times Sep 26, 2019

breznak commented Sep 26, 2019

View reviewed changes

breznak closed this Sep 26, 2019

breznak reopened this Sep 26, 2019

Eye/Retina Encoder - progress.

914d46b

breznak added 15 commits September 28, 2019 11:24

Merge branch 'master_community' into retina_encoder

69d6953

Retina: fix imports

3edddad

some imports for Eye/Retina encoder needed fixing for newer versions

Eye: improve documentation

ce1e68b

Eye: remove resulution_factor as argument

2eace9e

make it a const variable

Eye: rm arg fovea_scale

692aec9

keep it only as member variable

Eye: add argument mode: both/parvo/magno

b803886

which retinal pathway to simulate

Eye: fix parvo:magno cells ratio

de65135

should be 1/3rd, not 3rd-root

Eye: color vs B/W mode

e2014ea

Eye: argument plot=False

f259ec6

let's you disable/enable plotting, useful for headless mode

Eye: remove image_file member

57fd3b0

self.image is used for both string/data functionality

Eye: remove EyeSensorSampler class as unneeded

f014438

Eye: fix parvo/magno split mode

45da86c

Revert "Eye: argument plot=False"

97e6f57

This reverts commit f259ec6.

Eye: comments

2cc80e1

Eye: compute accepts pos,rot,scale args

7ae83bf

breznak commented Oct 2, 2019

View reviewed changes

breznak requested a review from ctrl-z-9000-times October 2, 2019 13:19

breznak requested a review from ctrl-z-9000-times October 22, 2019 22:15

breznak mentioned this pull request Oct 22, 2019

Retina broken log polar WIP #721

Open

Eye: improve test

8025d82

breznak mentioned this pull request Oct 23, 2019

bioinspired.Retina: improve logSampling - custom center & diameter of fovea opencv/opencv_contrib#2303

Open

breznak added 5 commits October 23, 2019 14:11

Eye: provide dimensions, size

4e8b781

for compatibility with encoders

MNIST: use Retina image encoder WIP

854b450

Eye: fix dimensions

ede598d

Eye: fix test

f93c904

Eye: fixes for running only one of parvo/magno

aa3aea8

fixes in mnist + eye

2c94dbb

breznak added 7 commits October 23, 2019 19:04

Eye: improve main example

a063676

Eye: doc parameters

80747f5

`:

Eye: crop ROI to circle

2e50635

Eye: draw boundary around fovea

362ba32

in helper visualization

Eye: apply rotation before retina processing

fddbcba

Eye: use Retina's log sampling

565e69e

Eye: run example

116a91a

breznak commented Oct 23, 2019

View reviewed changes

breznak added 4 commits October 24, 2019 01:48

Eye: fix scaling

be40a5d

where plot was broken with frational scaling. Using cv2.resize() rather than numpy's roi.resize() fixes the issue (numerical problems)

Eye: bigger steps in random walk

d7c898e

eye: plot also the whole scene

b4100a2

Eye: compute has image is argument

3a6461c

breznak mentioned this pull request Oct 27, 2019

HTM tasks, capabilities - What you can do #683

Open

8 tasks

breznak mentioned this pull request Apr 14, 2020

How to encode large vector as input to SP and HTM #782

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Retina encoder: biological encoder for vision WIP #691

Retina encoder: biological encoder for vision WIP #691

breznak commented Sep 26, 2019 •

edited

breznak commented Sep 26, 2019

breznak left a comment •

edited

ctrl-z-9000-times commented Sep 27, 2019

breznak left a comment

breznak commented Oct 2, 2019

breznak commented Oct 22, 2019

breznak commented Oct 22, 2019 •

edited

breznak commented Oct 23, 2019

ctrl-z-9000-times commented Oct 23, 2019

breznak commented Oct 23, 2019

ctrl-z-9000-times commented Oct 23, 2019

breznak left a comment •

edited

breznak Oct 23, 2019

breznak Oct 23, 2019

breznak Oct 23, 2019

breznak Oct 23, 2019

breznak Oct 23, 2019

Retina encoder: biological encoder for vision WIP #691

Are you sure you want to change the base?

Retina encoder: biological encoder for vision WIP #691

Conversation

breznak commented Sep 26, 2019 • edited

breznak commented Sep 26, 2019

breznak left a comment • edited

Choose a reason for hiding this comment

ctrl-z-9000-times commented Sep 27, 2019

breznak left a comment

Choose a reason for hiding this comment

breznak commented Oct 2, 2019

breznak commented Oct 22, 2019

breznak commented Oct 22, 2019 • edited

breznak commented Oct 23, 2019

ctrl-z-9000-times commented Oct 23, 2019

breznak commented Oct 23, 2019

ctrl-z-9000-times commented Oct 23, 2019

breznak left a comment • edited

Choose a reason for hiding this comment

breznak Oct 23, 2019

Choose a reason for hiding this comment

breznak Oct 23, 2019

Choose a reason for hiding this comment

breznak Oct 23, 2019

Choose a reason for hiding this comment

breznak Oct 23, 2019

Choose a reason for hiding this comment

breznak Oct 23, 2019

Choose a reason for hiding this comment

breznak commented Sep 26, 2019 •

edited

breznak left a comment •

edited

breznak commented Oct 22, 2019 •

edited

breznak left a comment •

edited