[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of Amazon Rekognition #2716

nakane11 · 2022-07-23T16:16:46Z

Without this change, client.initiate_auth has failed with botocore.exceptions.NoCredentialsError: Unable to locate credentials.
This error is solved by using AWS SDK for Python, boto3.

Following changes accompanying:

Publish rects and class results for visualization
Update json files for JSK users, please download from here
Update description in documantation

cc: @iory

…boto3

…TimeSynchronizer option

… env

… visualization

…p.index_exp.

…n for sample

jsk_perception/node_scripts/aws_auto_checkin_app.py

k-okada

Let me clearfy the contents of this PR

fix typo and color convert, support approxymate_sync/queue_size in approxymate sync: thanks for fix!
support use_aws_detection/use_usb_cam/gui args in sample_aws_auto_checkin_app.launch: This is also great improvement. Do you have any comment on comparison between aws_face_detection and OpenCV's face detection, for example performance and speed? Please add that to document https://github.com/jsk-ros-pkg/jsk_recognition/blob/master/doc/jsk_perception/nodes/aws_auto_checkin_app.rst#sample
I think I wrongly used use_window args to both sample_aws_auto_checkin.launch and sample_aws_detect_faces.launch, may be you can remove that and use gui. Please also add doc="..." to gui.
Using draw_rects.py to publish result images: good visualization is always welcome.

I think above fixes can be merged immediately

--

Could you explan more about using recognition / dynamodb ?
a) I can not reproduce botocore.exceptions.NoCredentialsError: Unable to locate credentials error. Because of using old libraries?

$ pip freeze| grep boto
boto3==1.4.2
botocore==1.16.19

b) I am not familier with dynamodb, but how is it differnet from current auto_cehckin_apps? Do they have better performance then current checkin? or do they have different interfaces? for example, how to add new faces? currently we put the face image to auto-check-in-gapp-register... buckets. Or it is just a creanup API of old aws checkkin example?
Basically libraries alwyas update, and our time is limited. So we should follow new API only when we really need this and we concentrate on improving your own/unique idea.

c) why you need new aws.json file? if I need to do something on existing aws.json file, please let me know, we'd like to use same aws.json for all aws_ examples.

nakane11 · 2022-07-24T14:02:06Z

I think I wrongly used use_window args to both sample_aws_auto_checkin.launch and sample_aws_detect_faces.launch, may be you can remove that and use gui. Please also add doc="..." to gui.

I fixed it by 9a10b5d.

Do you have any comment on comparison between aws_face_detection and OpenCV's face detection, for example performance and speed?

aws_face_detection (1 ~ 50 Hz) seems to be slow compared with OpenCV's face detection (30 ~ 500 Hz), but the result is more accurate. OpenCV often incorrectly recognizes faces.
@iory Did you have any other intention on this?

a) I can not reproduce botocore.exceptions.NoCredentialsError: Unable to locate credentials error. Because of using old libraries?

I am using the same libraries (boto3==1.4.2 and botocore==1.16.19) and followed the instruction written in jsk doc,
but NoCredentialsError occurred from the beginning. I tried the former program with the old json files again and the result was same.

b) I am not familier with dynamodb, but how is it differnet from current auto_cehckin_apps? Do they have better performance then current checkin? or do they have different interfaces?

Since I cannot execute current checkin and aws_detect_faces has no error, I applied boto3 client to aws_auto_checkin in the same way.
search_faces_by_image() don't return person names, so @iory modified to use dynamodb containing names by a81c97c.
I checked the account has the dynamodb table, but don't know how to make it.

c) why you need new aws.json file? if I need to do something on existing aws.json file, please let me know, we'd like to use same aws.json for all aws_ examples.

This PR needs the change of aws_access_key_id and aws_secret_access_key and the addition of DynamodbTable, CollectionId and MaxFaces. aws_detect_faces can work with new aws.json file, too.

Anyway, the condition of this problem should be checked first.

iory · 2022-07-24T14:40:50Z

@k-okada

a) I can not reproduce botocore.exceptions.NoCredentialsError: Unable to locate credentials error. Because of using old libraries?

I can reproduce the error in docker environment (18.04).
Could you show me all results of pip freeze?

root@turbo:~/catkin_ws/src/jsk_recognition/jsk_perception/sample/data# roslaunch jsk_perception sample_aws_auto_checkin_app.launch use_window:=false
... logging to /root/.ros/log/d32ff9b0-0b5d-11ed-a5bc-a4ae12770dd6/roslaunch-turbo-24706.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://turbo:35051/

SUMMARY
========

CLEAR PARAMETERS
 * /aws_auto_checkin_app/

PARAMETERS
 * /aws_auto_checkin_app/env_path: /tmp/env.json
 * /aws_auto_checkin_app/use_window: False
 * /camera/camera_nodelet_manager/num_worker_threads: 4
 * /camera/depth_rectify_depth/interpolation: 0
 * /camera/depth_registered_rectify_depth/interpolation: 0
 * /face_detection/debug_view: False
 * /face_detection/eyes_cascade_name: /usr/share/opencv...
 * /face_detection/face_cascade_name: /usr/share/opencv...
 * /face_detection/queue_size: 3
 * /face_detection/use_camera_info: False
 * /rosdistro: melodic
 * /rosversion: 1.14.13
 * /use_sim_time: True

NODES
  /
    aws_auto_checkin_app (jsk_perception/aws_auto_checkin_app.py)
    camera_base_link (tf2_ros/static_transform_publisher)
    camera_base_link1 (tf2_ros/static_transform_publisher)
    camera_base_link2 (tf2_ros/static_transform_publisher)
    camera_base_link3 (tf2_ros/static_transform_publisher)
    face_detection (opencv_apps/face_detection)
    rosbag_play (rosbag/play)
  /camera/
    camera_nodelet_manager (nodelet/nodelet)
    depth_metric (nodelet/nodelet)
    depth_metric_rect (nodelet/nodelet)
    depth_points (nodelet/nodelet)
    depth_rectify_depth (nodelet/nodelet)
    depth_registered_hw_metric_rect (nodelet/nodelet)
    depth_registered_metric (nodelet/nodelet)
    depth_registered_rectify_depth (nodelet/nodelet)
    points_xyzrgb_hw_registered (nodelet/nodelet)
    rgb_rectify_color (nodelet/nodelet)
  /camera/depth_registered/
    republish (image_transport/republish)
  /camera/rgb/
    republish (image_transport/republish)

auto-starting new master
process[master]: started with pid [24716]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to d32ff9b0-0b5d-11ed-a5bc-a4ae12770dd6
process[rosout-1]: started with pid [24727]
started core service [/rosout]
process[rosbag_play-2]: started with pid [24732]
process[camera/rgb/republish-3]: started with pid [24736]
process[camera/depth_registered/republish-4]: started with pid [24742]
process[camera/camera_nodelet_manager-5]: started with pid [24748]
[ INFO] [1658673313.365420697]: Initializing nodelet with 4 worker threads.
process[camera/rgb_rectify_color-6]: started with pid [24754]
process[camera/depth_rectify_depth-7]: started with pid [24766]
process[camera/depth_metric_rect-8]: started with pid [24773]
process[camera/depth_metric-9]: started with pid [24780]
process[camera/depth_points-10]: started with pid [24802]
process[camera/depth_registered_rectify_depth-11]: started with pid [24809]
process[camera/points_xyzrgb_hw_registered-12]: started with pid [24816]
process[camera/depth_registered_hw_metric_rect-13]: started with pid [24823]
process[camera/depth_registered_metric-14]: started with pid [24830]
process[camera_base_link-15]: started with pid [24837]
process[camera_base_link1-16]: started with pid [24844]
process[camera_base_link2-17]: started with pid [24851]
process[camera_base_link3-18]: started with pid [24857]
process[face_detection-19]: started with pid [24863]
process[aws_auto_checkin_app-20]: started with pid [24870]
[INFO] [1658673315.277332, 0.000000]: ROS node initialized as /aws_auto_checkin_app
[INFO] [1658673315.278855, 0.000000]: Loading AutoCheckin env variables from /tmp/env.json
[INFO] [1658673317.392938, 1497924866.949040]: Initiate User Auth for k-okada@jsk.t.u-tokyo.ac.jp
Traceback (most recent call last):
  File "/root/catkin_ws/src/jsk_recognition/jsk_perception/node_scripts/aws_auto_checkin_app.py", line 165, in <module>
    auto = AutoCheckIn()
  File "/root/catkin_ws/src/jsk_recognition/jsk_perception/node_scripts/aws_auto_checkin_app.py", line 67, in __init__
    self.id_token = self._get_id_token_by_cognito(env['UserName'], env['UserPassword'])
  File "/root/catkin_ws/src/jsk_recognition/jsk_perception/node_scripts/aws_auto_checkin_app.py", line 37, in _get_id_token_by_cognito
    'PASSWORD': password
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 622, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/lib/python2.7/dist-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
[aws_auto_checkin_app-20] process has died [pid 24870, exit code 1, cmd /root/catkin_ws/src/jsk_recognition/jsk_perception/node_scripts/aws_auto_checkin_app.py image:=/camera/rgb/image_raw face_roi:=/face_detection/faces __name:=aws_auto_checkin_app __log:=/root/.ros/log/d32ff9b0-0b5d-11ed-a5bc-a4ae12770dd6/aws_auto_checkin_app-20.log].
log file: /root/.ros/log/d32ff9b0-0b5d-11ed-a5bc-a4ae12770dd6/aws_auto_checkin_app-20*.log
[ WARN] [1658673319.806117194, 1497924869.356813237]: '/face_detection' subscribes topics only with child subscribers.

k-okada · 2022-07-25T00:43:04Z

See https://gist.github.com/k-okada/b8f4fac48f8e473c4ae9328b916a0176 for the output of pip freeze and python -v ./aws_auto_checkin_app.py image:=/camera/rgb/image_raw face_roi:=/face_detection/faces __name:=aws_auto_checkin_app

k-okada · 2022-07-25T01:06:11Z

jsk_perception/node_scripts/aws_auto_checkin_app.py

        self.roi_sub = message_filters.Subscriber('face_roi', FaceArrayStamped)
-        self.ts = message_filters.ApproximateTimeSynchronizer([self.image_sub, self.roi_sub], 10, 1, allow_headerless = True)


please consider using

self.transport_hint = rospy.get_param( namespace + 'image_transport', 'compressed')

like
https://github.com/jsk-ros-pkg/coral_usb_ros/blob/2c5ddeb033a094795db927c31685fd7958104f8f/python/coral_usb/node_base.py#L128-L135

so that we can switch compressed image / raw image
c.f.
http://wiki.ros.org/image_transport/Tutorials/ExaminingImagePublisherSubscriber

Thank you. raw image becomes available by 0fdd1d8

iory · 2022-07-25T04:18:16Z

Thank you for the result of pip freeze.
I have reflected the results of your pip freeze in my environment in the following way.
cat freeze.log | xargs -n 1 pip install

Also, I created a simple test program.
https://gist.github.com/iory/6bc13df3db273a230ca0f669715302ea

However, the result was nocredentialserror.

root@zinn:/catkin_ws# python test.py
MD5SUM: fffde28eb16059813880ad603c265e61
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    'PASSWORD': password
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 622, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/lib/python2.7/dist-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

@k-okada
Could you show me the md5sum of env.json?

…nput raw image

k-okada · 2022-07-25T08:01:58Z

sorry, I had my own ~/.aws/credentials file and found current aws_auto_checkin_app.py need to load "aws.json". If we can use existing aws.json / env.json, then please revert link to google doc.

I also ask to change "env.json" as follows.

{
    "Region": "us-west-2",
    "UserName": "xxx",
    "UserPassword": "xxx"
    "FaceSimilarityThreshold": 90,
## For aws_auto_checkin_apps.py with rekognition/dynamodb   https://github.com/jsk-ros-pkg/jsk_recognition/pull/2716)
    "xxxx": "xxxx"
## For old aws_auto_checkin_apps.py (see https://github.com/jsk-ros-pkg/jsk_recognition/pull/2716)
    "ApiEndpoint" : "ykxgawtu49.execute-api.us-west-2.amazonaws.com/prod/rekognize_face",
    "CognitoUserPoolId": "us-west-2_lcp26Bg9R",
    "CognitoUserPoolClientId": "70bj4jrl8r6qs855sa5ft6o3uf",
    "FaceAreaThreshold": 1e4,
    "FaceMarginRatio": 0.2,
    "CroppedImageWidth": 540,
    "CroppedImageHeight": 540,
    "NameTtlSec": 10,
    "UseDeepLeaningForDetector": true,
}

iory · 2022-07-25T08:23:14Z

If we can use existing aws.json / env.json, then please revert link to google doc.

I confirmed that the old aws.json could not reference the names of individuals recognized using dynamodb.
The recognitionid and name corresponding to the person are registered in dynamodb.
We can see this from the following link.
https://us-west-2.console.aws.amazon.com/dynamodbv2/home?region=us-west-2#item-explorer?initialTagKey=&maximize=true&table=auto-check-in-app-DynamoDBTable-3ZGGIU72KJXH

Note that, the new aws.json is created by me from the aws console.
https://us-east-1.console.aws.amazon.com/iam/home?region=us-west-2#/security_credentials?credentials=iam

@nakane11

I also ask to change "env.json" as follows.

Could you change the env.json?

…t on use_aws_face_detection

nakane11 · 2022-07-25T09:04:05Z

I changed env.json and aws.json and updated the link in doc. Previous json files can be removed.

have mistaken

k-okada · 2022-07-25T10:14:21Z

hi, i have added AmazonDynamoDBReadONlyAccess polcy to rekognition users, so can we use old aws.json file? If so, I think we we want to keep using old files.

nakane11 · 2022-07-25T10:47:07Z

@k-okada
I confirmed that aws_auto_checkin_app worked with old aws.json file.
I restored the old aws.json to drive.

iory · 2022-07-25T11:04:51Z

I restored the old aws.json to drive.

Thanks. I deactivated my aws.json.

nakane11 and others added 12 commits July 23, 2022 13:06

[jsk_perception] Fix color space conversion

c04280a

[doc/jsk_perception] Fix topic name in documentation

e94a18f

[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of …

802677a

…boto3

[jsk_perception/aws_auto_checkin] Use ConnectionBasedNodelet and add …

0eec063

…TimeSynchronizer option

[jsk_perception/aws_auto_checkin] Use dynamodb to refer person name.

a81c97c

[jsk_perception/aws_auto_checkin] Add use_usb_cam to sample

c771cbb

[jsk_perception/aws_auto_checkin] Read ConnectionId and MaxFaces from…

df672ef

… env

[jsk_perception/aws_auto_checkin] Publish rects and class results for…

412b600

… visualization

Merge branch 'aws' of https://github.com/iory/jsk_recognition into aws

fc73f41

[jsk_perception/aws_auto_checkin] Fix aws_credentials_path

88fedb9

[jsk_perception/aws_auto_checkin] Update doc

3699bc5

[doc/aws_auto_checkin] Fix typo

2632df2

iory requested review from k-okada and iory July 23, 2022 16:24

iory added bug enhancement pkg/jsk_perception labels Jul 23, 2022

iory added 2 commits July 24, 2022 04:34

[jsk_perception/aws_auto_checkin] Fix invalid slice. To fix it, use n…

4219876

…p.index_exp.

[jsk_perception/aws_auto_checkin] Enable use_aws_face_detection optio…

a5b2ad6

…n for sample

nakane11 force-pushed the aws branch from d0c0809 to 5c8fc12 Compare July 23, 2022 23:31

[jsk_perception/auto_checkin_app] Fix documentation

256d621

nakane11 force-pushed the aws branch from 5c8fc12 to 256d621 Compare July 23, 2022 23:43

iory requested changes Jul 24, 2022

View reviewed changes

jsk_perception/node_scripts/aws_auto_checkin_app.py Show resolved Hide resolved

nakane11 added 2 commits July 24, 2022 15:19

[jsk_perception/aws_auto_checkin_app] import sys

9aa7797

[jsk_perception/aws_auto_checkin_app] Fix registerCallback

5259e01

k-okada reviewed Jul 24, 2022

View reviewed changes

[jsk_perception] Remove arg use_window

9a10b5d

k-okada reviewed Jul 25, 2022

View reviewed changes

[jsk_perception/aws_auto_checkin_app] Add transport_hint and enable i…

0fdd1d8

…nput raw image

[jsk_perception/aws_auto_checkin_app] Update json link and add commen…

bc08f89

…t on use_aws_face_detection

iory previously approved these changes Jul 25, 2022

View reviewed changes

iory approved these changes Jul 25, 2022

View reviewed changes

k-okada merged commit 19a8155 into jsk-ros-pkg:master Jul 25, 2022

nakane11 deleted the aws branch July 25, 2022 23:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of Amazon Rekognition #2716

[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of Amazon Rekognition #2716

nakane11 commented Jul 23, 2022 •

edited

k-okada left a comment •

edited

nakane11 commented Jul 24, 2022 •

edited

iory commented Jul 24, 2022

k-okada commented Jul 25, 2022 •

edited

k-okada Jul 25, 2022

nakane11 Jul 25, 2022

iory commented Jul 25, 2022

k-okada commented Jul 25, 2022

iory commented Jul 25, 2022

nakane11 commented Jul 25, 2022

k-okada commented Jul 25, 2022

nakane11 commented Jul 25, 2022

iory commented Jul 25, 2022

		self.roi_sub = message_filters.Subscriber('face_roi', FaceArrayStamped)
		self.ts = message_filters.ApproximateTimeSynchronizer([self.image_sub, self.roi_sub], 10, 1, allow_headerless = True)

[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of Amazon Rekognition #2716

[jsk_perception/aws_auto_checkin_app] Use search_faces_by_image() of Amazon Rekognition #2716

Conversation

nakane11 commented Jul 23, 2022 • edited

k-okada left a comment • edited

Choose a reason for hiding this comment

nakane11 commented Jul 24, 2022 • edited

iory commented Jul 24, 2022

k-okada commented Jul 25, 2022 • edited

k-okada Jul 25, 2022

Choose a reason for hiding this comment

nakane11 Jul 25, 2022

Choose a reason for hiding this comment

iory commented Jul 25, 2022

k-okada commented Jul 25, 2022

iory commented Jul 25, 2022

nakane11 commented Jul 25, 2022

k-okada commented Jul 25, 2022

nakane11 commented Jul 25, 2022

iory commented Jul 25, 2022

nakane11 commented Jul 23, 2022 •

edited

k-okada left a comment •

edited

nakane11 commented Jul 24, 2022 •

edited

k-okada commented Jul 25, 2022 •

edited