Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: process images/videos in parallel with multiprocessing #620

Merged
merged 6 commits into from
May 22, 2023

Conversation

ptpt
Copy link
Member

@ptpt ptpt commented May 22, 2023

❯ du -sh ~/Downloads/MyGoProVideos
 99G	/Users/taopeng/Downloads/MyGoProVideos

Before

❯ time python3 -m mapillary_tools.commands process ~/Downloads/MyGoProVideos/
Extracting geotags from images: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 954/954 [00:04<00:00, 196.96images/s]
Extracting GPS tracks from videos: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [03:53<00:00,  4.59s/videos]
2023-05-22 16:38:52,715 - INFO    - Validating 1005 metadatas
2023-05-22 16:39:00,045 - INFO    - Checking upload status for 1005 metadatas
2023-05-22 16:39:00,744 - INFO    - Check the description file for details: /Users/taopeng/Downloads/MyGoProVideos/mapillary_image_description.json
2023-05-22 16:39:00,750 - INFO    -       51 gopro(s) read in total
2023-05-22 16:39:00,750 - INFO    - 	       27 gopro(s) are ready to be uploaded
2023-05-22 16:39:00,750 - ERROR   - 	       22 gopro(s) failed due to MapillaryGPXEmptyError
2023-05-22 16:39:00,750 - ERROR   - 	        1 gopro(s) failed due to MapillaryStationaryVideoError
2023-05-22 16:39:00,750 - WARNING - 	        1 gopro(s) skipped due to MapillaryUploadedAlreadyError
2023-05-22 16:39:00,750 - INFO    -      954 image(s) read in total
2023-05-22 16:39:00,750 - INFO    - 	      954 image(s) are ready to be uploaded
2023-05-22 16:39:00,751 - ERROR   - MapillaryProcessError: Failed to process 23 files. To skip these errors, specify --skip_process_errors
python3 -m mapillary_tools.commands process ~/Downloads/MyGoProVideos/  234.23s user 7.40s system 97% cpu 4:07.52 total

Now

❯ time python3 -m mapillary_tools.commands process ~/Downloads/MyGoProVideos/
Extracting geotags from images: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 954/954 [00:01<00:00, 727.81images/s]
Extracting GPS tracks from videos: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:47<00:00,  1.08videos/s]
2023-05-22 16:41:11,687 - INFO    - Validating 1005 metadatas
2023-05-22 16:41:18,542 - INFO    - Checking upload status for 1005 metadatas
2023-05-22 16:41:19,130 - INFO    - Check the description file for details: /Users/taopeng/Downloads/MyGoProVideos/mapillary_image_description.json
2023-05-22 16:41:19,136 - INFO    -       51 gopro(s) read in total
2023-05-22 16:41:19,136 - INFO    - 	       27 gopro(s) are ready to be uploaded
2023-05-22 16:41:19,136 - ERROR   - 	       22 gopro(s) failed due to MapillaryGPXEmptyError
2023-05-22 16:41:19,136 - ERROR   - 	        1 gopro(s) failed due to MapillaryStationaryVideoError
2023-05-22 16:41:19,136 - WARNING - 	        1 gopro(s) skipped due to MapillaryUploadedAlreadyError
2023-05-22 16:41:19,136 - INFO    -      954 image(s) read in total
2023-05-22 16:41:19,136 - INFO    - 	      954 image(s) are ready to be uploaded
2023-05-22 16:41:19,136 - ERROR   - MapillaryProcessError: Failed to process 23 files. To skip these errors, specify --skip_process_errors
python3 -m mapillary_tools.commands process ~/Downloads/MyGoProVideos/  265.89s user 10.66s system 484% cpu 57.047 total

@ptpt ptpt merged commit 2507149 into main May 22, 2023
17 checks passed
@ptpt ptpt deleted the feat-multiprocessing-process branch May 22, 2023 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants