Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: move CdsApi to BuildSearchResult and HTTPDownload #1029

Merged
merged 21 commits into from
Apr 9, 2024

Conversation

dalpasso
Copy link
Collaborator

@dalpasso dalpasso commented Feb 20, 2024

New BuildSearchResult plugin to replace CdsApi with HTTPDownload and GenericAuth for providers cop_ads and cop_cds.
Order mechanism configured in place of previous internal CdsApi handling.

@dalpasso dalpasso linked an issue Feb 20, 2024 that may be closed by this pull request
@dalpasso dalpasso self-assigned this Feb 20, 2024
@dalpasso dalpasso force-pushed the 995-handle-cdsapi-download-retries-in-eodag branch from 14fc3de to e5183e9 Compare February 28, 2024 10:25
@dalpasso dalpasso marked this pull request as ready for review February 28, 2024 10:25
Copy link
Contributor

github-actions bot commented Feb 28, 2024

Test Results

    4 files  ± 0      4 suites  ±0   5m 27s ⏱️ -33s
  504 tests  -  3    501 ✅  -  3   3 💤 ±0  0 ❌ ±0 
2 016 runs   - 12  1 930 ✅  - 12  86 💤 ±0  0 ❌ ±0 

Results for commit b84dcca. ± Comparison against base commit 8090f54.

This pull request removes 9 and adds 6 tests. Note that renamed tests count towards both.
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_authenticate
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_discover_queryables
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_download
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_download_all
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_logging
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_query_dates_missing
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_query_with_custom_producttype
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_query_with_producttype
tests.units.test_apis_plugins.TestApisPluginCdsApi ‑ test_plugins_apis_cds_query_without_producttype
tests.units.test_download_plugins.TestDownloadPluginHttp ‑ test_plugins_download_http_order_download_cds
tests.units.test_search_plugins.TestSearchPluginBuildSearchResult ‑ test_plugins_search_buildsearchresult_dates_missing
tests.units.test_search_plugins.TestSearchPluginBuildSearchResult ‑ test_plugins_search_buildsearchresult_discover_queryables
tests.units.test_search_plugins.TestSearchPluginBuildSearchResult ‑ test_plugins_search_buildsearchresult_with_custom_producttype
tests.units.test_search_plugins.TestSearchPluginBuildSearchResult ‑ test_plugins_search_buildsearchresult_with_producttype
tests.units.test_search_plugins.TestSearchPluginBuildSearchResult ‑ test_plugins_search_buildsearchresult_without_producttype

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Feb 28, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      318      27  91.51%   80-82, 91, 99, 103-105, 178, 190, 391-393, 457-460, 507-508, 517-518, 597, 666-671, 673
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    727      76  89.55%   87-96, 369, 582, 626-629, 667, 772, 776-781, 807, 877, 954, 1068-1073, 1159-1171, 1211, 1213, 1217, 1238-1240, 1244-1255, 1268-1274, 1364-1367, 1396-1416, 1464, 1470-1473, 1482, 1848, 1881-1887, 2152, 2156-2159, 2173-2175, 2210
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        211      32  84.83%   53-60, 64-66, 170-177, 261-262, 352, 388, 449, 463-466, 479, 503-506, 549-555
api/product/metadata_mapping.py                649      82  87.37%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 348-360, 401-404, 441, 462-465, 488, 496-497, 570-571, 595-596, 602-605, 620-621, 770, 816, 933-938, 1069, 1083-1103, 1123, 1128, 1238, 1260, 1274, 1287-1306, 1345, 1397, 1435-1439, 1458
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       3  86.96%   25, 48, 55
plugins/manager.py                             127      14  88.98%   49-51, 95-100, 146, 185, 205, 231, 270-271
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       2  89.47%   26, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 81      13  83.95%   35-37, 86, 99, 101, 114-117, 165-168
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      10  80.00%   32-34, 51-52, 71, 80-83, 85, 92-95
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        491     165  66.40%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 458-459, 465-469, 502, 537, 541, 548, 578-586, 590, 628-636, 643-645, 686-760, 778-839, 850-855, 871-884, 913, 928-930, 933, 943-951, 959-972, 982-1001, 1008-1020, 1061, 1087, 1132-1134, 1354
plugins/download/base.py                       261      57  78.16%   58-64, 145, 180, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       443      93  79.01%   82-88, 122, 194-206, 278-283, 307, 309, 373, 400-402, 412, 418, 420, 444, 483-487, 544, 617-673, 691, 724-733, 759-760, 787, 811, 819-824, 842-843, 850, 911-917, 972-973, 979, 989, 1055, 1059, 1073-1089
plugins/download/s3rest.py                     117      27  76.92%   55-58, 124, 165, 199, 229-236, 239-241, 245, 258-264, 272-273, 276-280, 303, 324-327
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py          178      22  87.64%   62, 97, 142-143, 149, 160, 289-292, 326, 383-395, 457, 460, 470, 487, 515, 517
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          198      65  67.17%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     560      64  88.57%   89, 361-367, 375-376, 483-489, 544-547, 620-621, 662, 680, 695, 748, 769, 772-773, 782, 793, 802, 825, 885-890, 894-895, 923, 994, 1013-1030, 1069, 1145-1146, 1210-1211, 1232, 1257, 1263, 1294-1295, 1305-1311, 1354, 1368, 1388, 1478
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   169      10  94.08%   242, 513, 515, 518-520, 592, 599-603
rest/server.py                                 298      54  81.88%   80-81, 105, 128-129, 240-242, 258, 298-299, 311-327, 409-414, 444, 601-608, 639, 682-683, 771-773, 790-795, 825, 827, 831-832, 836-837
rest/stac.py                                   436      95  78.21%   59-61, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 659-660, 840, 905-907, 1126, 1136-1148, 1161-1183, 1197-1242, 1401-1402
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185       9  95.14%   52-55, 231-235, 288, 291, 359
rest/types/stac_queryables.py                   30       2  93.33%   28, 114
rest/types/stac_search.py                      122      10  91.80%   48-51, 167, 182-184, 192, 196
rest/utils/__init__.py                         116      14  87.93%   53, 108-109, 128-130, 180, 190-204, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       3  90.32%   78, 90, 92
types/__init__.py                               76       6  92.11%   53, 87, 152, 172, 177, 185
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              495      43  91.31%   83, 88, 109-111, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1098, 1182, 1200, 1375
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8648    1524  82.38%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
config.py                                   +3       0  +0.08%
plugins/download/http.py                   +11      -8  +2.39%
plugins/search/build_search_result.py     +120     +17  -3.74%
TOTAL                                     +134      +9  +0.32%

Results for commit: b84dcca

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Feb 28, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                     11       0  100.00%
cli.py                                         300      47  84.33%   61, 647-686, 788-839, 843
config.py                                      318      28  91.19%   80-82, 91, 99, 103-105, 178, 190, 391-393, 457-460, 507-508, 517-518, 597, 631, 666-671, 673
crunch.py                                        6       6  0.00%    18-24
api/__init__.py                                  1       0  100.00%
api/core.py                                    727      76  89.55%   87-96, 369, 582, 626-629, 667, 772, 776-781, 807, 877, 954, 1068-1073, 1159-1171, 1211, 1213, 1217, 1238-1240, 1244-1255, 1268-1274, 1364-1367, 1396-1416, 1464, 1470-1473, 1482, 1848, 1881-1887, 2152, 2156-2159, 2173-2175, 2210
api/search_result.py                            44       6  86.36%   33-35, 70, 79, 86, 100
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          44       5  88.64%   27-29, 79, 129
api/product/_product.py                        211      32  84.83%   53-60, 64-66, 170-177, 261-262, 352, 388, 449, 463-466, 479, 503-506, 549-555
api/product/metadata_mapping.py                649      83  87.21%   66-68, 129-131, 232, 264-265, 317-329, 331, 342, 348-360, 401-404, 441, 462-465, 488, 496-497, 570-571, 595-596, 602-605, 620-621, 770, 816, 933-938, 1069, 1083-1103, 1123, 1128, 1238, 1260, 1274, 1287-1306, 1345, 1397, 1420, 1435-1439, 1458
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      8       2  75.00%   23, 41
plugins/__init__.py                              1       0  100.00%
plugins/base.py                                 23       4  82.61%   25, 48, 55, 68
plugins/manager.py                             127      14  88.98%   49-51, 95-100, 146, 185, 205, 231, 270-271
plugins/apis/__init__.py                         1       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           99      15  84.85%   47-55, 156-158, 205-206, 232-234
plugins/apis/usgs.py                           172      36  79.07%   59-64, 128, 201, 235, 270-272, 277, 303-304, 309, 339-346, 357-362, 384-390, 392-398, 421
plugins/authentication/__init__.py               7       1  85.71%   31
plugins/authentication/aws_auth.py              20       2  90.00%   25-27
plugins/authentication/base.py                  19       3  84.21%   26, 34, 47
plugins/authentication/generic.py               16       3  81.25%   28, 40, 50
plugins/authentication/header.py                17       1  94.12%   27
plugins/authentication/keycloak.py              88      17  80.68%   32-34, 159-160, 190-212, 238-243
plugins/authentication/oauth.py                 15       8  46.67%   25, 32-34, 38-41
plugins/authentication/openid_connect.py       103      64  37.86%   39-41, 138-150, 154-172, 180-222, 228-237, 246-286, 291-299, 304-305
plugins/authentication/qsauth.py                36       2  94.44%   32, 83
plugins/authentication/sas_auth.py              49       2  95.92%   32, 76
plugins/authentication/token.py                 81      13  83.95%   35-37, 86, 99, 101, 114-117, 165-168
plugins/crunch/__init__.py                       1       0  100.00%
plugins/crunch/base.py                          10       2  80.00%   25, 38
plugins/crunch/filter_date.py                   62      15  75.81%   30, 53-58, 72, 81, 90, 93, 105-107, 116-118, 125
plugins/crunch/filter_latest_intersect.py       50      35  30.00%   32-34, 48-53, 69-114
plugins/crunch/filter_latest_tpl_name.py        33       2  93.94%   28, 86
plugins/crunch/filter_overlap.py                68      17  75.00%   28-30, 33, 82-85, 91, 99, 110-126
plugins/crunch/filter_property.py               33       8  75.76%   29, 60-65, 68-69, 85-89
plugins/download/__init__.py                     1       0  100.00%
plugins/download/aws.py                        491     165  66.40%   77-83, 272, 285, 352-355, 369-373, 419-421, 425, 458-459, 465-469, 502, 537, 541, 548, 578-586, 590, 628-636, 643-645, 686-760, 778-839, 850-855, 871-884, 913, 928-930, 933, 943-951, 959-972, 982-1001, 1008-1020, 1061, 1087, 1132-1134, 1354
plugins/download/base.py                       261      59  77.39%   58-64, 145, 180, 250-252, 319-320, 340-346, 377-381, 387-388, 432, 435-449, 461, 465, 538-542, 572-573, 581-598, 605-613, 615-619, 666, 689, 711, 719
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       443      93  79.01%   82-88, 122, 194-206, 278-283, 307, 309, 373, 400-402, 412, 418, 420, 444, 483-487, 544, 617-673, 691, 724-733, 759-760, 787, 811, 819-824, 842-843, 850, 911-917, 972-973, 979, 989, 1055, 1059, 1073-1089
plugins/download/s3rest.py                     117      27  76.92%   55-58, 124, 165, 199, 229-236, 239-241, 245, 258-264, 272-273, 276-280, 303, 324-327
plugins/search/__init__.py                       1       0  100.00%
plugins/search/base.py                         127       9  92.91%   49-54, 108, 112, 275, 295, 378
plugins/search/build_search_result.py          178      29  83.71%   62, 97, 142-143, 149, 160, 289-292, 326, 383-395, 457, 460, 470, 487, 507-522
plugins/search/creodias_s3.py                   54       3  94.44%   55, 73, 107
plugins/search/csw.py                          107      83  22.43%   43-45, 57-58, 62-63, 74-122, 128-141, 149-181, 199-240
plugins/search/data_request_search.py          198      65  67.17%   52, 89-92, 108, 120, 124-125, 139, 144, 149, 156, 169-172, 226-227, 231, 241-247, 252, 280-283, 291-302, 319, 321, 328-329, 331-332, 350-354, 387, 394, 405, 418, 424-436, 441
plugins/search/qssearch.py                     560      88  84.29%   89, 361-367, 375-376, 483-489, 544-547, 620-621, 662, 680, 695, 748, 769, 772-773, 782, 793, 802, 825, 885-890, 894-895, 923, 994, 1013-1030, 1069, 1145-1146, 1210-1211, 1232, 1257, 1263, 1294-1295, 1305-1311, 1354, 1368, 1388, 1438-1507
plugins/search/static_stac_search.py            47       3  93.62%   39-40, 82
rest/__init__.py                                 1       0  100.00%
rest/core.py                                   169      61  63.91%   140, 142, 144, 150-151, 168-176, 182-188, 234-263, 417-444, 462, 512-551, 592, 599-603
rest/server.py                                 298     298  0.00%    18-850
rest/stac.py                                   436     154  64.68%   59-61, 214, 229-231, 273, 286-295, 314-320, 365, 402-404, 427, 462-463, 549-594, 639, 647-648, 652-660, 782, 840, 905-907, 924-926, 934-936, 949-951, 965-982, 992-1013, 1023-1045, 1053-1070, 1093-1116, 1126, 1136-1148, 1161-1183, 1197-1242, 1395-1421
rest/types/__init__.py                           1       0  100.00%
rest/types/eodag_search.py                     185      18  90.27%   52-55, 231-235, 268-270, 288, 291, 297, 301, 359, 371-374
rest/types/stac_queryables.py                   30       6  80.00%   28, 53-58, 114
rest/types/stac_search.py                      122      12  90.16%   48-51, 167, 182-184, 192, 196, 240, 243
rest/utils/__init__.py                         116      31  73.28%   53, 79-85, 105, 108-109, 128-130, 147, 173-181, 188-209, 236
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           31       5  83.87%   73-74, 78, 90, 92
types/__init__.py                               76      11  85.53%   53, 87, 125, 152, 162-164, 172, 177, 185, 195
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 104
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              495      44  91.11%   83, 88, 109-111, 189-190, 199-226, 229, 243, 325-329, 405-409, 430-432, 514, 519, 529, 567-568, 964-967, 975-976, 1017-1018, 1098, 1181-1182, 1200, 1375
utils/constraints.py                           123      42  65.85%   31, 84-93, 134, 139, 143, 154, 177-178, 189-197, 206, 220-236, 245-256
utils/exceptions.py                             37       2  94.59%   23, 93
utils/import_system.py                          30      20  33.33%   27, 67-81, 93-103
utils/logging.py                                29       1  96.55%   123
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/stac_reader.py                            91      28  69.23%   55-56, 63-86, 93-95, 99, 141, 155-158
TOTAL                                         8648    1980  77.10%

Diff against develop

Filename                                 Stmts    Miss  Cover
-------------------------------------  -------  ------  -------
config.py                                   +3       0  +0.08%
plugins/download/http.py                   +11      -8  +2.39%
plugins/search/build_search_result.py     +120     +24  -7.67%
utils/__init__.py                            0      +1  -0.20%
TOTAL                                     +134     +17  +0.25%

Results for commit: b84dcca

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@sbrunato sbrunato marked this pull request as draft April 5, 2024 14:27
@sbrunato sbrunato self-assigned this Apr 5, 2024
@sbrunato sbrunato force-pushed the 995-handle-cdsapi-download-retries-in-eodag branch from ef3bf78 to a9fcf9e Compare April 8, 2024 20:04
@sbrunato sbrunato force-pushed the 995-handle-cdsapi-download-retries-in-eodag branch from a9fcf9e to 813013a Compare April 8, 2024 20:10
@sbrunato sbrunato marked this pull request as ready for review April 8, 2024 20:11
@sbrunato sbrunato force-pushed the 995-handle-cdsapi-download-retries-in-eodag branch from 813013a to b84dcca Compare April 8, 2024 20:13
@sbrunato sbrunato changed the title feat: generate download link for CdsApi plugin internally to EODAG refactor: move CdsApi to BuildSearchResult and HTTPDownload Apr 9, 2024
@sbrunato sbrunato merged commit b3e40f6 into develop Apr 9, 2024
10 checks passed
@sbrunato sbrunato deleted the 995-handle-cdsapi-download-retries-in-eodag branch April 9, 2024 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

handle CdsApi download retries in eodag
2 participants