Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(providers): wekeo split to wekeo_main and wekeo_ecmwf #1214

Merged
merged 19 commits into from
Aug 9, 2024

Conversation

anesson-cs
Copy link
Collaborator

@anesson-cs anesson-cs commented Jun 14, 2024

Split wekeo provider into wekeo_main and wekeo_ecmwf (and already separated wekeo_cmems) to separate various data (Copernicus Sentinel, DEM, and CLMS data) and ECMWF data which are queried differently and that have two different way to discover their product types queryables.
All these wekeo_* providers are available through a wekeo providers group.

@sbrunato sbrunato marked this pull request as draft June 14, 2024 15:14
Copy link
Contributor

github-actions bot commented Jun 14, 2024

Test Results

    4 files  ± 0      4 suites  ±0   6m 21s ⏱️ +10s
  559 tests + 4    556 ✅ + 4    3 💤 ±0  0 ❌ ±0 
2 236 runs  +16  2 134 ✅ +16  102 💤 ±0  0 ❌ ±0 

Results for commit 32a4b05. ± Comparison against base commit fad1677.

♻️ This comment has been updated with latest results.

Copy link
Contributor

github-actions bot commented Jun 14, 2024

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         303      50  83.50%   654-700, 802-853, 857
config.py                                      367      27  92.64%   82-84, 93, 101, 105-107, 175, 186, 452-454, 514-517, 560-561, 570-571, 676, 739-744, 746
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    768      73  90.49%   634, 678-681, 719, 763, 797, 850-855, 881, 972, 1040, 1170, 1253-1265, 1301, 1303, 1331, 1335-1346, 1359-1365, 1448-1451, 1484-1504, 1556, 1573-1576, 1588-1591, 1613-1620, 1934, 1965-1971, 2222, 2226-2229, 2240-2242, 2274
api/search_result.py                            51       4  92.16%   73, 82, 89, 103
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                675      81  88.00%   129-131, 226, 258-259, 305-306, 316-328, 330, 341, 347-359, 404-405, 442, 463-466, 489, 497-498, 574-575, 599-600, 606-609, 624-625, 774, 820, 991-996, 1123, 1137-1157, 1177, 1182, 1289, 1311, 1325, 1338-1357, 1396, 1448, 1480-1484, 1499
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             130      12  90.77%   104-109, 159, 197, 219, 223, 247, 281-282
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   34, 47
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       4  91.30%   132, 156-161
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       183      17  90.71%   119, 133-158, 166, 320-323, 347
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 88      16  81.82%   79, 107, 109, 131-143, 198-202
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   51-52, 68, 77-80, 82, 89-92
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        489     163  66.67%   265, 278, 345-348, 362-366, 408-410, 414, 446-447, 453-457, 486, 518, 522, 529, 559-567, 571, 603-611, 618-620, 654-728, 746-807, 818-823, 835-848, 873, 888-890, 893, 903-911, 919-932, 942-973, 980-992, 1030, 1056, 1101-1103, 1323
plugins/download/base.py                       253      51  79.84%   136, 164, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       534     129  75.84%   195-207, 209-210, 242-245, 306-309, 311-312, 319-324, 342-357, 374-376, 388, 436, 443-449, 467, 481, 495, 503-505, 521-526, 537, 555, 597-601, 623, 663, 708, 722-728, 757-821, 839, 872-881, 903-904, 931-936, 942, 945, 961, 978-979, 1009-1010, 1017, 1078-1084, 1139-1140, 1146, 1156, 1192, 1228, 1246-1262, 1288-1290
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         128      13  89.84%   102, 106, 117, 255, 275, 331-332, 352, 355-363
plugins/search/build_search_result.py          181      23  87.29%   95, 136-137, 143, 154, 288-291, 320, 373-390, 450, 453, 463, 480, 508, 510
plugins/search/cop_marine.py                   197      47  76.14%   55, 63-65, 71-72, 88, 90, 93, 128-130, 142-143, 183-192, 196, 199, 203, 221, 249, 253, 268, 272, 276, 280, 284-288, 294-297, 300-314, 331, 354, 357, 363
plugins/search/creodias_s3.py                   55       3  94.55%   53, 71, 105
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          200      67  66.50%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-324, 326-327, 345-349, 382, 392, 403, 416, 422-437, 442
plugins/search/qssearch.py                     681      90  86.78%   388, 392-398, 406-407, 513-519, 563, 579, 589, 608-623, 660-663, 737-738, 786, 805, 812, 824, 881, 902, 905-906, 915-916, 925-926, 935-936, 963, 1034-1039, 1043-1052, 1086, 1108, 1168, 1217, 1291-1295, 1355, 1358, 1364-1365, 1386, 1413-1425, 1432, 1464-1466, 1476-1482, 1512, 1535, 1550, 1572, 1655-1658, 1663-1666, 1693-1694, 1709
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  26       0  100.00%
rest/constants.py                                7       0  100.00%
rest/core.py                                   234      26  88.89%   269, 355, 454, 670, 677-725
rest/server.py                                 283      53  81.27%   108, 131-133, 246-248, 304-305, 317-333, 425-430, 458, 623-630, 659, 703-704, 727, 798-800, 817-822, 851, 853, 857-858, 862-863
rest/stac.py                                   464     120  74.14%   303, 325, 373-376, 400-426, 431-437, 457-459, 482, 514-515, 543, 556, 597-637, 659-675, 749-762, 769, 823-824, 885, 950-952, 1156, 1164-1176, 1187-1209, 1220-1265, 1424-1425
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     179       5  97.21%   232-236, 289, 292, 360
rest/types/queryables.py                        56       1  98.21%   164
rest/types/stac_search.py                      126       7  94.44%   129, 175, 190-192, 200, 204
rest/utils/__init__.py                          93      12  87.10%   108-109, 128-130, 182, 192-206
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              114      14  87.72%   53, 70, 129-132, 199, 213-222, 232, 253, 266
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              500      37  92.60%   85, 90, 194-195, 204-231, 234, 248, 328-332, 406-410, 429-431, 510, 525, 561-562, 926-929, 937-938, 976-977, 1148
utils/constraints.py                           119      37  68.91%   89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             35       1  97.14%   93
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                29       1  96.55%   121
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 88, 90, 92, 94, 96, 112, 120-122, 130
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9578    1636  82.92%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  -------
api/core.py                     +10      -1  +0.25%
plugins/search/base.py            0      +4  -3.13%
plugins/search/qssearch.py      +15      +9  -1.06%
TOTAL                           +25     +12  -0.08%

Results for commit: 32a4b05

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

Copy link
Contributor

github-actions bot commented Jun 14, 2024

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         303      50  83.50%   654-700, 802-853, 857
config.py                                      367      28  92.37%   82-84, 93, 101, 105-107, 175, 186, 452-454, 514-517, 560-561, 570-571, 676, 707, 739-744, 746
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    768      73  90.49%   634, 678-681, 719, 763, 797, 850-855, 881, 972, 1040, 1170, 1253-1265, 1301, 1303, 1331, 1335-1346, 1359-1365, 1448-1451, 1484-1504, 1556, 1573-1576, 1588-1591, 1613-1620, 1934, 1965-1971, 2222, 2226-2229, 2240-2242, 2274
api/search_result.py                            51       4  92.16%   73, 82, 89, 103
api/product/__init__.py                          6       0  100.00%
api/product/_assets.py                          48       5  89.58%   75, 147, 155, 158-162
api/product/_product.py                        187      20  89.30%   70-72, 237-238, 313, 342, 399, 413-416, 429, 453-456, 499-505
api/product/metadata_mapping.py                675      81  88.00%   129-131, 226, 258-259, 305-306, 316-328, 330, 341, 347-359, 404-405, 442, 463-466, 489, 497-498, 574-575, 599-600, 606-609, 624-625, 774, 820, 991-996, 1123, 1137-1157, 1177, 1182, 1289, 1311, 1325, 1338-1357, 1396, 1448, 1480-1484, 1499
api/product/drivers/__init__.py                  6       0  100.00%
api/product/drivers/base.py                      6       1  83.33%   38
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             130      12  90.77%   104-109, 159, 197, 219, 223, 247, 281-282
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           91       8  91.21%   149-151, 198-199, 225-227
plugins/apis/usgs.py                           180      31  82.78%   132, 234, 268, 303-305, 310, 336-337, 342, 372-379, 390-395, 417-423, 425-431, 454
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              19       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   34, 47
plugins/authentication/generic.py               14       2  85.71%   40, 50
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       4  91.30%   132, 156-161
plugins/authentication/oauth.py                 13       7  46.15%   32-34, 38-41
plugins/authentication/openid_connect.py       183      17  90.71%   119, 133-158, 166, 320-323, 347
plugins/authentication/qsauth.py                34       1  97.06%   83
plugins/authentication/sas_auth.py              47       1  97.87%   76
plugins/authentication/token.py                 88      16  81.82%   79, 107, 109, 131-143, 198-202
plugins/authentication/token_exchange.py        35      19  45.71%   74-80, 92-120
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   40
plugins/crunch/filter_date.py                   59      14  76.27%   49-54, 66, 75, 84, 87, 99-101, 110-112, 119
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   48-53, 66-111
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 68-71, 78-81, 87, 95, 106-122
plugins/crunch/filter_property.py               30       7  76.67%   54-59, 62-63, 79-83
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        489     163  66.67%   265, 278, 345-348, 362-366, 408-410, 414, 446-447, 453-457, 486, 518, 522, 529, 559-567, 571, 603-611, 618-620, 654-728, 746-807, 818-823, 835-848, 873, 888-890, 893, 903-911, 919-932, 942-973, 980-992, 1030, 1056, 1101-1103, 1323
plugins/download/base.py                       253      53  79.05%   136, 164, 231-233, 296-297, 314-320, 351-355, 361-362, 404, 407-421, 433, 437, 501-505, 535-536, 544-561, 568-576, 578-582, 625, 647, 669, 677
plugins/download/creodias_s3.py                 17       9  47.06%   44-58
plugins/download/http.py                       534     130  75.66%   195-207, 209-210, 242-245, 306-309, 311-312, 319-324, 342-357, 374-376, 388, 436, 443-449, 467, 481, 495, 503-505, 521-526, 537, 555, 597-601, 623, 663, 708, 722-728, 757-821, 839, 872-881, 903-904, 931-936, 942, 945, 961, 978-979, 992, 1009-1010, 1017, 1078-1084, 1139-1140, 1146, 1156, 1192, 1228, 1246-1262, 1288-1290
plugins/download/s3rest.py                     116      24  79.31%   113, 149, 156, 191, 221-228, 231-233, 237, 248-254, 262-263, 266-270, 293, 314-317
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         128      14  89.06%   102, 106, 117, 255, 275, 331-332, 352, 355-363, 365
plugins/search/build_search_result.py          181      30  83.43%   95, 136-137, 143, 154, 288-291, 320, 373-390, 450, 453, 463, 480, 500-515
plugins/search/cop_marine.py                   197      47  76.14%   55, 63-65, 71-72, 88, 90, 93, 128-130, 142-143, 183-192, 196, 199, 203, 221, 249, 253, 268, 272, 276, 280, 284-288, 294-297, 300-314, 331, 354, 357, 363
plugins/search/creodias_s3.py                   55       3  94.55%   53, 71, 105
plugins/search/csw.py                          105      81  22.86%   58-59, 63-64, 72-120, 126-139, 147-179, 197-238
plugins/search/data_request_search.py          200      67  66.50%   90-93, 109, 120, 124-125, 136, 141, 146, 153, 166-169, 223-224, 228, 238-244, 249, 275-278, 286-297, 314, 316, 323-324, 326-327, 345-349, 382, 392, 403, 416, 422-437, 442
plugins/search/qssearch.py                     681     119  82.53%   388, 392-398, 406-407, 513-519, 563, 566, 579, 589, 608-623, 660-663, 737-738, 786, 805, 812, 824, 881, 902, 905-906, 915-916, 925-926, 935-936, 963, 1034-1039, 1043-1052, 1086, 1108, 1168, 1217, 1291-1295, 1355, 1358, 1364-1365, 1386, 1413-1425, 1432, 1464-1466, 1476-1482, 1512, 1535, 1550, 1572, 1638-1738
plugins/search/static_stac_search.py            72      10  86.11%   98-125, 141, 154
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  26       4  84.62%   34-36, 68
rest/constants.py                                7       0  100.00%
rest/core.py                                   234     149  36.32%   160-241, 260-314, 327-365, 396-428, 443-459, 477-488, 497-533, 550, 592-631, 670, 677-725
rest/server.py                                 283     283  0.00%    18-874
rest/stac.py                                   464     393  15.30%   123-129, 136-161, 187-192, 219, 236-359, 369-438, 453-492, 505-558, 566-589, 597-637, 659-675, 685, 699-702, 718-788, 804-840, 863-888, 895-911, 918-923, 932-959, 968-970, 977-979, 991-993, 1005-1022, 1030-1051, 1059-1081, 1088-1105, 1124-1147, 1156, 1164-1176, 1187-1209, 1220-1265, 1272-1289, 1297-1455
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     179      15  91.62%   122, 232-236, 269-271, 289, 292, 298, 302, 360, 372-375
rest/types/queryables.py                        56      13  76.79%   51-52, 59-60, 67-68, 94-99, 108-109, 164
rest/types/stac_search.py                      126      18  85.71%   127-129, 153-154, 159-160, 175, 190-192, 200, 204, 251-256
rest/utils/__init__.py                          93      30  67.74%   79-85, 105, 108-109, 128-130, 143, 150, 175-183, 190-211
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       6  73.91%   40, 43-44, 48, 60, 62
types/__init__.py                              114      39  65.79%   53, 66-70, 81-93, 120-122, 129-132, 172, 199, 209-225, 230, 232, 253, 258, 266, 276
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             81       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
types/whoosh.py                                 15       0  100.00%
utils/__init__.py                              500      39  92.20%   85, 90, 194-195, 204-231, 234, 248, 328-332, 406-410, 429-431, 510, 525, 561-562, 926-929, 937-938, 976-977, 1006, 1148, 1309
utils/constraints.py                           119      37  68.91%   89-98, 139, 144, 148, 159, 182-184, 194, 208-224, 233-244
utils/exceptions.py                             35       1  97.14%   93
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                29       1  96.55%   121
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   30       8  73.33%   36, 38, 42, 76, 94-101
utils/requests.py                               55      11  80.00%   64, 88, 90, 92, 94, 96, 112, 120-122, 130
utils/rest.py                                   36       1  97.22%   55
utils/stac_reader.py                           111      45  59.46%   56-57, 63-85, 95-97, 101, 137, 153-156, 203-212, 222-252
TOTAL                                         9578    2429  74.64%

Diff against develop

Filename                      Stmts    Miss  Cover
--------------------------  -------  ------  -------
api/core.py                     +10      -1  +0.25%
plugins/search/qssearch.py      +15     +10  -1.10%
TOTAL                           +25      +9  -0.03%

Results for commit: 32a4b05

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@anesson-cs anesson-cs marked this pull request as ready for review June 14, 2024 18:26
@sbrunato sbrunato marked this pull request as draft June 19, 2024 05:57
@anesson-cs anesson-cs marked this pull request as ready for review June 21, 2024 17:28
@alambare alambare requested a review from sbrunato June 26, 2024 07:56
@alambare alambare requested review from dalpasso and removed request for sbrunato July 18, 2024 08:59
docs/getting_started_guide/providers.rst Outdated Show resolved Hide resolved
docs/plugins.rst Outdated Show resolved Hide resolved
docs/notebooks/tutos/tuto_wekeo.ipynb Outdated Show resolved Hide resolved
eodag/plugins/search/qssearch.py Outdated Show resolved Hide resolved
eodag/plugins/search/qssearch.py Outdated Show resolved Hide resolved
tests/units/test_search_plugins.py Outdated Show resolved Hide resolved
tests/units/test_search_plugins.py Outdated Show resolved Hide resolved
tests/units/test_search_plugins.py Outdated Show resolved Hide resolved
docs/plugins.rst Outdated Show resolved Hide resolved
@sbrunato sbrunato changed the title Split wekeo provider feat(providers): wekeo_main and wekeo_ecmwf providers Aug 9, 2024
@sbrunato sbrunato changed the title feat(providers): wekeo_main and wekeo_ecmwf providers feat(providers): wekeo split to wekeo_main and wekeo_ecmwf Aug 9, 2024
@sbrunato sbrunato merged commit e864f53 into develop Aug 9, 2024
10 checks passed
@sbrunato sbrunato deleted the split-wekeo-provider branch August 9, 2024 07:32
@sbrunato sbrunato added this to the 3.0.0.dev milestone Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants