DELF codebase general cleanup #9930

andrefaraujo · 2021-04-20T23:02:18Z

Description

Some general cleanup of the DELF codebase, including:

Cleanup after PRs Dataset utilities added. #9796 and Global model supporting functions #9901
Fix old imports delf.python.detect_to_retrieve import dataset to point to the correct new location
Remove “app” import from TF and use directly the one from “absl” package
Some other remaining general cleanup, eg to standardize formatting

Type of change

For a new feature or function, please create an issue first to discuss it
with us before submitting a pull request.

Note: Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)

Tests

Existing tests now all pass.

Test Configuration:

Checklist

I have signed the Contributor License Agreement.
I have read guidelines for pull request.
My code follows the coding guidelines.
I have performed a self code review of my own code.
I have commented my code, particularly in hard-to-understand areas.
I have made corresponding changes to the documentation.
My changes generate no new warnings.
I have added tests that prove my fix is effective or that my feature works.

253126424 by Andre Araujo: Scripts to compute metrics for Google Landmarks dataset. Also, a small fix to metric in retrieval case: avoids duplicate predicted images. -- 253118971 by Andre Araujo: Metrics for Google Landmarks dataset. -- 253106953 by Andre Araujo: Library to read files from Google Landmarks challenges. -- 250700636 by Andre Araujo: Handle case of aggregation extraction with empty set of input features. -- 250516819 by Andre Araujo: Add minimum size for DELF extractor. -- 250435822 by Andre Araujo: Add max_image_size/min_image_size for open-source DELF proto / module. -- 250414606 by Andre Araujo: Refactor extract_aggregation to allow reuse with different datasets. -- 250356863 by Andre Araujo: Remove unnecessary cmd_args variable from boxes_and_features_extraction. -- 249783379 by Andre Araujo: Create directory for writing mapping file if it does not exist. -- 249581591 by Andre Araujo: Refactor scripts to extract boxes and features from images in Revisited datasets. Also, change tf.logging.info --> print for easier logging in open source code. -- 249511821 by Andre Araujo: Small change to function for file/directory handling. -- 249289499 by Andre Araujo: Internal change. -- PiperOrigin-RevId: 253126424

PiperOrigin-RevId: 253135576

Merging upstream to origin.

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

PiperOrigin-RevId: 253248648

PiperOrigin-RevId: 260619120

…ting to dataset library. PiperOrigin-RevId: 260998608

Also adds a unit test for previously-existing SaveMetricsFile function. PiperOrigin-RevId: 263361410

PiperOrigin-RevId: 264437080

PiperOrigin-RevId: 265127245

PiperOrigin-RevId: 277177468

PiperOrigin-RevId: 287316435

PiperOrigin-RevId: 288842237

PiperOrigin-RevId: 289487669

309118395 by Andre Araujo: Make DELF open-source code compatible with TF2. -- 309067582 by Andre Araujo: Handle image resizing rounding properly for python extraction. New behavior is tested with unit tests. -- 308690144 by Andre Araujo: Several changes to improve DELF model/training code and make it work in TF 2.1.0: - Rename some files for better clarity - Using compat.v1 versions of functions - Formatting changes - Using more appropriate TF function names -- 308689397 by Andre Araujo: Internal change. -- 308341315 by Andre Araujo: Remove old slim dependency in DELF open-source model. This avoids issues with requiring old TF-v1, making it compatible with latest TF. -- 306777559 by Andre Araujo: Internal change -- 304505811 by Andre Araujo: Raise error during geometric verification if local features have different dimensionalities. -- 301739992 by Andre Araujo: Transform some geometric verification constants into arguments, to allow custom matching. -- 301300324 by Andre Araujo: Apply name change(experimental_run_v2 -> run) for all callers in Tensorflow. -- 299919057 by Andre Araujo: Automated refactoring to make code Python 3 compatible. -- 297953698 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 297521242 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 297278247 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 297270405 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 297238741 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 297108605 by Andre Araujo: Explicitly replace "import tensorflow" with "tensorflow.compat.v1" for TF2.x migration -- 294676131 by Andre Araujo: Add option to resize images to square resolutions without aspect ratio preservation. -- 293849641 by Andre Araujo: Internal change. -- 293840896 by Andre Araujo: Changing Slim import to tf_slim codebase. -- 293661660 by Andre Araujo: Allow the delf training script to read from TFRecords dataset. -- 291755295 by Andre Araujo: Internal change. -- 291448508 by Andre Araujo: Internal change. -- 291414459 by Andre Araujo: Adding train script. -- 291384336 by Andre Araujo: Adding model export script and test. -- 291260565 by Andre Araujo: Adding placeholder for Google Landmarks dataset. -- 291205548 by Andre Araujo: Definition of DELF model using Keras ResNet50 as backbone. -- 289500793 by Andre Araujo: Add TFRecord building script for delf. -- PiperOrigin-RevId: 309118395

Merging with upstream.

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

327279047 by Andre Araujo: Adapt extractor to handle new form of joint local+global extraction. -- 326733524 by Andre Araujo: Internal change. -- PiperOrigin-RevId: 327279047

Merging upstream.

328982978 by Andre Araujo: Updated DELG model training so that the size of the output tensor is unchanged by the GeM pooling layer. Export global model trained with DELG global features. -- 328218938 by Andre Araujo: Internal change. -- PiperOrigin-RevId: 328982978

Merging upstream.

330022709 by Andre Araujo: Export joint local+global TF2 DELG model, and enable such joint extraction. Also, rename export_model.py -> export_local_model.py for better clarity. To check that the new exporting code is doing the right thing, I compared features extracted from the new exported model against those extracted from models exported with a single modality, using the same checkpoint. They are identical. Some other small changes: - small automatic reformating - small documentation improvements -- PiperOrigin-RevId: 330022709

Merging upstram.

331625297 by Andre Araujo: Internal change. -- 330062115 by Andre Araujo: Fix small (non-critical) typo in the DELG extractor. -- PiperOrigin-RevId: 331625297

Merging.

347479009 by Andre Araujo: Fix image size setting for GLD training. -- PiperOrigin-RevId: 347479009

Merging to upstream.

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_0/google3/..

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_1/google3/..

andrefaraujo · 2021-04-20T23:07:31Z

@ykate1998 Kate, FYI here's the PR I've done after your recent mergings. One thing I'd like to ask: can you please check in the tensorboard-related utils that you had in the models/research/delf/delf/python/training/global_features_utils.py file into a new separate file named models/research/delf/delf/python/training/tensorboard_utils.py ? I removed them from global_features_utils.py since it simplifies things internally a lot.

andrefaraujo · 2021-04-22T23:25:34Z

@dan-anghel Friendly ping, could you take a look at this soon? :)

dan-anghel

Apologies for the delay @andrefaraujo. Looks good! Thank you!

andrefaraujo added 30 commits June 13, 2019 16:35

Updating DELF init to adjust to latest changes

3153738

Editing init files for python packages

f53250e

Edit D2R dataset reader to work with py3.

d321b1b

PiperOrigin-RevId: 253135576

DELF package: fix import ordering

3624965

Merge remote-tracking branch 'upstream/master'

608bd22

Merging upstream to origin.

Adding new requirements to setup.py

d23506d

Adding init file for training dir

4f9756d

Merged commit includes the following changes:

f054e3f

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

Adding init file for training subdirs

6026239

Working version of DELF training

f95f991

Internal change.

6dc7f9a

PiperOrigin-RevId: 253248648

Fix variance loading in open-source code.

4519efe

PiperOrigin-RevId: 260619120

Separate image re-ranking as a standalone library, and add metric wri…

64f45bd

…ting to dataset library. PiperOrigin-RevId: 260998608

Tool to read written D2R Revisited datasets metrics file. Test is added.

620dc62

Also adds a unit test for previously-existing SaveMetricsFile function. PiperOrigin-RevId: 263361410

Add optional resize factor for feature extraction.

49116a9

PiperOrigin-RevId: 264437080

Fix NumPy's new version spacing changes.

bb799f9

PiperOrigin-RevId: 265127245

Maker image matching function visible, and add support for RANSAC seed.

ec3d60b

PiperOrigin-RevId: 277177468

Avoid matplotlib failure due to missing display backend.

7d1f100

PiperOrigin-RevId: 287316435

Removes tf.contrib dependency.

59b7fef

PiperOrigin-RevId: 288842237

Fix tf contrib removal for feature_aggregation_extractor.

0bc84d4

PiperOrigin-RevId: 289487669

Merge remote-tracking branch 'upstream/master'

a56b550

Merging with upstream.

Updating README, dependency versions

b50913d

Updating training README

189432f

Fixing init import of export_model

5b54e56

Fixing init import of export_model_utils

18d5f78

tkinter in INSTALL_INSTRUCTIONS

05e7804

Merged commit includes the following changes:

ce8ef85

FolderOrigin-RevId: /google/src/cloud/andrearaujo/delf_oss/google3/..

INSTALL_INSTRUCTIONS mentioning different cloning options

4879ffa

andrefaraujo added 23 commits August 23, 2020 16:38

Merged commit includes the following changes:

4fb12b5

327279047 by Andre Araujo: Adapt extractor to handle new form of joint local+global extraction. -- 326733524 by Andre Araujo: Internal change. -- PiperOrigin-RevId: 327279047

Updated DELG instructions after model extraction refactoring

19e42d5

Updating GLDv2 paper model baseline

6d24570

Merge remote-tracking branch 'upstream/master'

3a2d764

Merging upstream.

Updated training README after recent changes

f91c181

Updated training README to fix small typo

88d20a0

Merge remote-tracking branch 'upstream/master'

8d14533

Merging upstream.

Updated DELG exporting instructions

e0ef6a6

Updated DELG exporting instructions: fix small typo

07f1b84

Merge remote-tracking branch 'upstream/master'

8c0e759

Merging upstram.

Adding DELG pre-trained models on GLDv2-clean

8da1db5

Merged commit includes the following changes:

723b3ab

331625297 by Andre Araujo: Internal change. -- 330062115 by Andre Araujo: Fix small (non-critical) typo in the DELG extractor. -- PiperOrigin-RevId: 331625297

Merge remote-tracking branch 'upstream/master'

7a82d81

Merging.

Merge remote-tracking branch 'upstream/master'

7935626

Merging.

Merged commit includes the following changes:

9dcd445

347479009 by Andre Araujo: Fix image size setting for GLD training. -- PiperOrigin-RevId: 347479009

Merge remote-tracking branch 'upstream/master'

c3fb284

Merging to upstream.

Merged commit includes the following changes:

f65a869

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_0/google3/..

Merged commit includes the following changes:

447a6ff

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_0/google3/..

Merge to upstream

a2e3e88

Merged commit includes the following changes:

2a1ada8

FolderOrigin-RevId: /google/src/cloud/andrearaujo/copybara_25C283E7A3474256A7C206FC5ABF7E8D_1/google3/..

Add whiten module import

9d55fe6

andrefaraujo requested a review from dan-anghel April 20, 2021 23:02

google-cla bot added the cla: yes label Apr 20, 2021

dan-anghel approved these changes Apr 23, 2021

View reviewed changes

andrefaraujo merged commit 8a0f227 into tensorflow:master Apr 23, 2021

andrefaraujo mentioned this pull request Apr 23, 2021

Global model training #9904

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

DELF codebase general cleanup #9930

DELF codebase general cleanup #9930

Uh oh!

andrefaraujo commented Apr 20, 2021

Uh oh!

andrefaraujo commented Apr 20, 2021

Uh oh!

andrefaraujo commented Apr 22, 2021

Uh oh!

dan-anghel left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

DELF codebase general cleanup #9930

DELF codebase general cleanup #9930

Uh oh!

Conversation

andrefaraujo commented Apr 20, 2021

Description

Type of change

Tests

Checklist

Uh oh!

andrefaraujo commented Apr 20, 2021

Uh oh!

andrefaraujo commented Apr 22, 2021

Uh oh!

dan-anghel left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants