Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regression testing spa #210

Closed
wants to merge 8 commits into from
Closed

Regression testing spa #210

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
4683d2d
test for tag-up
spa0002 Aug 30, 2022
e1da4d0
Update test_checks_files.py
spa0002 Aug 30, 2022
cbfffb0
update test_check_files, checks_dif10_master_test_file.json
spa0002 Sep 1, 2022
2d9998b
Update test_check_files.py
spa0002 Sep 13, 2022
00d9b5e
Updates to test files in tests/ folder
spa0002 Sep 15, 2022
d804237
updates to custom_validator, rule_mapping, dif10 test file, test_chec…
spa0002 Sep 15, 2022
384850c
Update test_check_files.py
spa0002 Sep 15, 2022
9da7c21
Update checks_dif10_master_test_file.json
spa0002 Sep 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file modified .DS_Store
View file
Open in desktop
Binary file not shown.
Binary file modified pyQuARC/.DS_Store
View file
Open in desktop
Binary file not shown.
6 changes: 3 additions & 3 deletions pyQuARC/code/custom_validator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ def __init__(self):
super().__init__()

@staticmethod
@if_arg
def ends_at_present_flag_logic_check(
ends_at_present_flag, ending_date_time, collection_state
):
Expand Down Expand Up @@ -62,7 +61,7 @@ def availability_check(field_value, parent_value):
@if_arg
def bounding_coordinate_logic_check(west, north, east, south):
# Checks if the logic for coordinate values make sense
result = {"valid": False, "value": ""}
result = {"valid": False, "value": [west, north, east, south]}
west = float(west)
east = float(east)
south = float(south)
Expand Down Expand Up @@ -138,8 +137,9 @@ def user_services_check(first_name, middle_name, last_name):

@staticmethod
def doi_missing_reason_explanation(explanation, missing_reason, doi):
validity = bool(doi or ((not doi) and missing_reason and explanation))
return {
"valid": doi or not missing_reason or explanation,
"valid": validity,
"value": explanation
}

Expand Down
6 changes: 4 additions & 2 deletions pyQuARC/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,21 @@

from tqdm import tqdm

from code.utils import get_headers
# from code.utils import get_headers # test

if __name__ == '__main__':
from code.utils import get_headers
from code.checker import Checker
from code.constants import COLOR, ECHO10_C, SUPPORTED_FORMATS
from code.downloader import Downloader
from code.utils import get_cmr_url, is_valid_cmr_url
else:
from .code.utils import get_headers
from .code.checker import Checker
from .code.constants import COLOR, ECHO10_C, SUPPORTED_FORMATS
from .code.downloader import Downloader
from .code.utils import get_cmr_url, is_valid_cmr_url


ABS_PATH = os.path.abspath(os.path.dirname(__file__))
END = COLOR["reset"]
Expand Down
2 changes: 1 addition & 1 deletion pyQuARC/schemas/MimeType.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"Hits: 37","page_num: 1","page_size: 2000","Keyword Version: 13.8","Revision: 2022-06-03 10:18:29","Timestamp: 2022-06-28 10:40:09","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/MimeType/?format=xml","Case native"
"Hits: 37","page_num: 1","page_size: 2000","Keyword Version: 14.3","Revision: 2022-08-26 10:35:56","Timestamp: 2022-09-15 12:28:30","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/MimeType/?format=xml","Case native"
MimeType,UUID
"application/gml+xml","40bdf6e5-780c-43e2-ab8e-e5dfae4bd779"
"application/gzip","a8ee535a-8bc8-46fd-8b97-917bd7ea7666"
Expand Down
2 changes: 1 addition & 1 deletion pyQuARC/schemas/chronounits.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"Hits: 184","page_num: 1","page_size: 2000","Keyword Version: 13.8","Revision: 2022-06-03 10:13:23","Timestamp: 2022-06-28 09:05:58","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/chronounits/?format=xml","Case native"
"Hits: 184","page_num: 1","page_size: 2000","Keyword Version: 14.3","Revision: 2022-08-26 10:32:48","Timestamp: 2022-09-15 10:12:53","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/chronounits/?format=xml","Case native"
Eon,Era,Period,Epoch,Age,Sub-Age,UUID
"ARCHAEAN","EOARCHEAN","","","","","8dff752e-939d-4b16-a735-bb4af8dbd785"
"ARCHAEAN","MESOARCHEAN","","","","","27660bb3-982e-414c-991a-707abd861f46"
Expand Down
22 changes: 13 additions & 9 deletions pyQuARC/schemas/granuledataformat.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
"Hits: 121","page_num: 1","page_size: 2000","Keyword Version: 13.8","Revision: 2022-06-03 10:18:40","Timestamp: 2022-06-28 10:40:06","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/DataFormat/?format=xml","Case native"
"Hits: 125","page_num: 1","page_size: 2000","Keyword Version: 14.3","Revision: 2022-08-26 10:36:02","Timestamp: 2022-09-15 10:12:56","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/DataFormat/?format=xml","Case native"
Short_Name,Long_Name,UUID
"ACCDB","Microsoft Access ACCDB File Format Family","378d394c-8230-49f1-8ba0-8ee7e7abe316"
"ADF","ArcInfo Binary Grid Format","81782805-cc0d-4325-8f35-aa0c8e439b16"
"AGF","Atlas Geo File","067315ec-ed74-401f-b828-d568351211d1"
"AMES","","08208d5b-1311-4c14-b8c1-5f64f75cb392"
"AMES","NASA Ames Format for Data Exchange","08208d5b-1311-4c14-b8c1-5f64f75cb392"
"AREA","McIDAS Image File Format","ebe6f3c4-a78f-4152-977c-296d42e4e9e8"
"ASCII Grid","Esri American Standard Code for Information Interchange Grid","d7d5b771-8c3e-44d5-ba14-8a3757cecbaf"
"ASCII Raster","Esri American Standard Code for Information Interchange Raster","1aee02b9-bf37-4197-bb7f-bd6f040d26f6"
Expand All @@ -22,6 +22,7 @@ Short_Name,Long_Name,UUID
"CCSDS","Consultative Committee for Space Data Systems","2da9aa88-c3d4-4307-a86f-e048b6297899"
"CEOS","Committee on Earth Observation Satellites","c0158436-501c-47e5-9a92-6416ef81d0b9"
"COG","Cloud Optimized GeoTIFF","23a0f833-fc2b-4922-8998-371a4a18bd17"
"CR2","Canon Raw Version 2","87ae4923-1a96-4fa2-a560-ddf27de4fcba"
"CRD","Consolidated Laser Ranging Data Format","49028622-39d1-46b1-b89f-0fc2b4923882"
"CSV","Comma-Separated Values File","465809cc-e76c-4630-8594-bb8bd7a1a380"
"DBF","dBASE Database File","c087e039-7d05-4067-a0e3-150e0ff19aa3"
Expand Down Expand Up @@ -54,8 +55,9 @@ Short_Name,Long_Name,UUID
"ICARTT","International Consortium for Atmospheric Research on Transport and Transformation","23ccdce5-cd51-424e-b82a-67e076a86994"
"ICI","","0679d78d-0931-4948-94ec-46ab130785a6"
"IFC","Industry Foundation Classes","5513aef2-5676-4a19-9652-edda4d753c2e"
"IIQ","Intelligent Image Quality","28acdf05-9aed-44e8-a58c-e47291bbc28f"
"IONEX","IONEX Format Data","133c2aee-50f9-4260-b792-6d6694399da3"
"ISI","","43e7a701-2ce2-41e8-8893-6b75963fcfe0"
"ISI","Indian Standards Institute","43e7a701-2ce2-41e8-8893-6b75963fcfe0"
"IWRF","Integrated Weather Radar Facility Time Series Format","e1c88c22-0bf2-4798-89fd-fa7d875e4b6c"
"JPEG2000","Joint Photographic Experts Group Format 2000","e1544d27-a3b8-4b14-95aa-e25b21bcce1f"
"JPEG","Joint Photographic Experts Group Format","7443bb2d-1dbb-44d1-bd29-0241d30fbc57"
Expand All @@ -68,12 +70,13 @@ Short_Name,Long_Name,UUID
"Little-Endian","","5cd15e09-8082-4049-98ae-f402e54929a4"
"MAT","MATLAB","10de1987-5896-42d6-be7c-506fd7ba1f21"
"MDB","Microsoft Access MDB File Format Family","1b5f24f0-a524-4d6b-a20a-6bc685d46c8e"
"MOV","QuickTime Moviev","8f05ce85-385b-491a-abe5-a39f426e4832"
"MOV","QuickTime Movie","8f05ce85-385b-491a-abe5-a39f426e4832"
"MP4","MPEG-4","a593731a-ff49-47d8-bed0-7f3f9d3a9e33"
"MSR","MetaSensing Rawdata","a1edd41d-50e1-4d1f-9062-7d2649605e38"
"McIDAS","Man computer Interactive Data Access System","77f3bc44-4f05-432d-9ee3-2e6859ca2896"
"NBJ","U.S. Military Campgrounds Directory Software","5c71f6b1-6978-416d-a213-f42a7d2728e3"
"NIDS","National Indicator Data Set","a82ccef5-a7bf-416c-8ef6-704e62564901"
"NITF21NCDRD","NITF (NCDRD) Raster Type","20c82be1-789f-4441-9728-d51cf2704523"
"Not Provided","","c7c7819e-4805-4b0a-819e-82b453e8fdd0"
"ODB","OpenDocument Database Front End Document Format","7f6d6202-7fb2-4ad5-bbfe-114803316b63"
"ODS","OpenDocument Spreadsheet Document Format","a2904083-1e32-4086-b028-a2afc2ffd443"
Expand All @@ -83,7 +86,7 @@ Short_Name,Long_Name,UUID
"PSD","Photoshop Format","5e17145e-b1c9-473b-9e5e-7680c286c64a"
"Parquet","Column-oriented Data Storage Format","cbcdfa99-4403-4370-a8d1-03b2327a51aa"
"PowerPoint","Microsoft PowerPoint","ceb2056a-ab97-441b-9491-62df68970205"
"RB5","","20f86520-af22-4e57-9d83-06a7e2ca5280"
"RB5","Qualcomm Flight RB5 5G","20f86520-af22-4e57-9d83-06a7e2ca5280"
"RData","R Data Format Family","53ce2fee-84fe-40c4-bd26-b2dcce86021a"
"RINEX","Receiver Independent Exchange Format","48571017-0cc3-4ac7-b9ab-d0df8ed99a6c"
"SAFE","Standard Archive Format for Europe","87eddebb-2ef4-4597-bfac-1b97e9f54440"
Expand All @@ -100,19 +103,20 @@ Short_Name,Long_Name,UUID
"Sea-Bird CTD","Sea-Bird for Conductivity, Temperature, and Depth","ddcd5941-9ba8-4c39-a498-1e214d1bfa6e"
"SeaBASS","SeaWiFS Bio-Optical Archive and Storage System Data Format","be0d9b66-445d-4109-8784-63f1ea80e729"
"Shapefile","Esri Shapefile","a1d1fbdd-98f4-41e9-9aeb-a369a3b466a4"
"SonTek Castaway CTD","","8b7ff128-106e-4a40-9de9-5bf3b6404fc2"
"SonTek Castaway CTD","Castaway Conductivity, Temperature, and Depth","8b7ff128-106e-4a40-9de9-5bf3b6404fc2"
"TAR","Tape Archive","19139079-41cd-47d5-b581-93bed04ade29"
"TIFF","Tagged Image File Format","0225ee3e-c3b1-4a5d-bd22-b36462330b00"
"Text File","","7de32ba8-eb3a-4e02-add3-3d828e46bd57"
"TimeseriesML","","80323a44-3233-4472-b0d9-158cb9371719"
"TimeseriesML","Timeseries Profile of Observations and Measurements","80323a44-3233-4472-b0d9-158cb9371719"
"UF","Universal Format","f8c1fd07-20b6-47fe-a420-f01679c523d1"
"VPF","Vector Product Format","75ce5e67-f1c3-4a53-bad3-34ba6f104a8c"
"Valeport CTD","","1528fc97-9aa0-4417-887b-c746fb0c6d23"
"Valeport CTD","Valeport Conductivity, Temperature, and Depth","1528fc97-9aa0-4417-887b-c746fb0c6d23"
"WKI","Lotus 1-2-3 Spreadsheet","5d95e94b-95f7-4db3-a555-a1363fc23df0"
"WKT","Well-known text","5aedf9cd-c6cb-4c62-b54f-050549e4a270"
"WaterML","Water Markup Language","8b3ff8b7-92b4-48ba-97dd-29e925069745"
"Word","Microsoft Word","f49d1197-2ed0-4f88-98ce-fbb86bbb03e1"
"XML","Extensible Markup Language","9efbc54a-b45c-47af-bac7-20e504483dc4"
"XTDR","","83698059-cf52-4267-9b2c-599f671d1bd6"
"XTDR","External Standard Data Representation","83698059-cf52-4267-9b2c-599f671d1bd6"
"YAML","YAML Ain't Markup Language","de132e9b-fc00-4c79-8a65-57f774d9a673"
"Zarr","Chunked Compressed N-dimensional Arrays","ed9804d8-e1ad-4209-8fea-30fba3d47ed7"
"miniSEED","mini Standard for the Exchange of Earthquake Data","3b7e3e2e-8f14-4e07-a84c-00dc9750cc10"
Expand Down
2 changes: 1 addition & 1 deletion pyQuARC/schemas/horizontalresolutionrange.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"Hits: 15","page_num: 1","page_size: 2000","Keyword Version: 13.8","Revision: 2022-06-03 10:17:38","Timestamp: 2022-06-28 09:06:03","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/horizontalresolutionrange/?format=xml","Case native"
"Hits: 15","page_num: 1","page_size: 2000","Keyword Version: 14.3","Revision: 2022-08-26 10:35:31","Timestamp: 2022-09-15 10:12:59","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/horizontalresolutionrange/?format=xml","Case native"
Horizontal_Resolution_Range,UUID
"1 km - < 10 km or approximately .01 degree - < .09 degree","6dd8224f-944e-4798-ac48-44c23a567eeb"
"1 meter - < 30 meters","abf43d91-a65d-4b3b-a6dd-593e211b2c7b"
Expand Down
2 changes: 1 addition & 1 deletion pyQuARC/schemas/idnnode.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"Hits: 127","page_num: 1","page_size: 2000","Keyword Version: 13.8","Revision: 2022-06-03 10:17:33","Timestamp: 2022-06-28 10:01:49","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/idnnode/?format=xml","Case native"
"Hits: 127","page_num: 1","page_size: 2000","Keyword Version: 14.3","Revision: 2022-08-26 10:35:28","Timestamp: 2022-09-15 08:36:36","Terms Of Use: https://cdn.earthdata.nasa.gov/conduit/upload/5182/KeywordsCommunityGuide_Baseline_v1_SIGNED_FINAL.pdf","The most up to date XML representations can be found here: https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/idnnode/?format=xml","Case native"
Short_Name,Long_Name,UUID
"ACADIS","","913d42e2-1641-4f5e-8273-379ddd3812d5"
"ACE/CRC","","13381733-968f-4fd6-b70c-aac6a77b2657"
Expand Down
Loading