updated docs & tools

- updated spot-data tool - updated docs
open-oceans · Nov 13, 2023 · 09f3bc5 · 09f3bc5
1 parent 7092858
commit 09f3bc5
Show file tree

Hide file tree

Showing 10 changed files with 111 additions and 119 deletions.
diff --git a/docs/installation.md b/docs/installation.md
@@ -27,18 +27,22 @@ As usual, to print help:
 
 ```
 pyspotter -h
-usage: pyspotter [-h] {auth,reset,devlist,spot-check,spot-data} ...
+
+usage: pyspotter [-h] {readme,auth,reset,devlist,spot-check,spot-data,snapshot,snapshot-latest} ...
 
 Simple CLI for Sofarocean API
 
 positional arguments:
-  {auth,reset,devlist,spot-check,spot-data}
+  {readme,auth,reset,devlist,spot-check,spot-data,snapshot,snapshot-latest}
+    readme              Go to the web based pyspotter cli readme page
     auth                Authenticates and saves your API token
     reset               Regenerates your API token
     devlist             Print lists of devices available under your account
     spot-check          Spot check a Spotter location and time
     spot-data           Export Spotter Data based on Spotter ID & grouped by date
+    snapshot            Saves the last 14 day data for a single or global spotters
+    snapshot-latest     Saves the latest wind/wave data from global spotters
 
-optional arguments:
+options:
   -h, --help            show this help message and exit
 ```
diff --git a/docs/projects/auth.md b/docs/projects/auth.md
@@ -4,4 +4,4 @@ This tool allows you to save your authentication token, this is then used for au
 
 ``` pyspotter auth```
 
-![pyspotter_auth](https://user-images.githubusercontent.com/6677629/147421243-6ca937c4-9614-42ae-9b49-82b3b2d4e286.gif)
+![pyspotter_auth](https://github.com/open-oceans/pyspotter/assets/6677629/ad97c5ec-4b8f-463a-bfb9-5682cfacf74a)
diff --git a/docs/projects/device-export.md b/docs/projects/device-export.md
@@ -2,7 +2,7 @@
 
 This tool was designed to get the datasets out of the spotter. It seems that API currently limited temporal data, and the best way to group seemed to be using dates. This script uses the result JSON objects, and adds a date field from the timestamp to make the grouping easy, since timestamps are unique. This then writes these CSV file with column headers and can export both wind and wave data as needed.
 
-![pyspotter_spot-data](https://user-images.githubusercontent.com/6677629/147421473-c3833f2b-8e0e-4188-af88-dd19f30eb74d.gif)
+![pyspotter_spotdata](https://github.com/open-oceans/pyspotter/assets/6677629/78f1a1d7-febc-4c97-9e93-b6a99355f6bd)
 
 ```
 usage: pyspotter spot-data [-h] --sid SID --dtype DTYPE --folder FOLDER

diff --git a/docs/projects/devlist.md b/docs/projects/devlist.md
@@ -2,7 +2,7 @@
 
 This tool will simply print the names of all devices to which you have access, instead of trying to remember the list. This tool requires no user input.
 
-![pyspotter_devices](https://user-images.githubusercontent.com/6677629/147421382-138a03b9-d2e1-4f55-92be-15f88e1ac9e5.gif)
+![pyspotter_devlist](https://github.com/open-oceans/pyspotter/assets/6677629/2fef1c74-f35d-4105-983b-582401ed59a3)
 
 ```
 usage: pyspotter devlist [-h]

diff --git a/docs/projects/reset.md b/docs/projects/reset.md
@@ -4,4 +4,4 @@ For some reason if you need to reset your token , this will allow you to use you
 
 ```pyspotter reset```
 
-![pyspotter_reset](https://user-images.githubusercontent.com/6677629/147421249-f2a7ceeb-7d24-41dd-bb50-6bef30913dbc.gif)
+![pyspotter_reset](https://github.com/open-oceans/pyspotter/assets/6677629/8cd8fd28-769b-4f05-a7ea-fec447b2a612)
diff --git a/docs/projects/snapshot.md b/docs/projects/snapshot.md
@@ -0,0 +1,19 @@
+# Global biweekly Snapshot
+
+This tool is designed to fetch a 14 day rolling biweekly snapshot of global spotter database. This can accept a single spotter id or can simply take the export folder to create the global snapshot CSV file. An extra column with system:time_start is added where the timestamp is converted to epoch time in milliseconds. Current export only includes wave data.
+
+![pyspotter_snapshot](https://github.com/open-oceans/pyspotter/assets/6677629/7875c21e-6e84-41e7-b246-7ccaa84d18ef)
+
+```
+pyspotter snapshot -h
+usage: pyspotter snapshot [-h] --export EXPORT [--sid SID]
+
+options:
+  -h, --help       show this help message and exit
+
+Required named arguments.:
+  --export EXPORT  Full path to folder to export results
+
+Optional named arguments:
+  --sid SID        Spotter ID
+```
diff --git a/docs/projects/spot-check.md b/docs/projects/spot-check.md
@@ -2,7 +2,7 @@
 
 This tool is built to fetch simply the latest information from the spotter including battery, humidity, power and lat long. Since these spotter can move across multiple time zones, it uses the lat long to estimate the time zone and converts the UTC time to local time for the spotter.
 
-![pyspotter_check](https://user-images.githubusercontent.com/6677629/147421383-6b8f16a0-b3ab-481f-ad2d-b3be15407b34.gif)
+![pyspotter_spotcheck](https://github.com/open-oceans/pyspotter/assets/6677629/cad35989-fd95-40eb-bd7d-bd58dfb56c18)
 
 ```
 pyspotter spot-check -h
@@ -19,5 +19,5 @@ Required named arguments.:
 Example usage would be
 
 ```
-pyspotter spot-check --sid 0320
+pyspotter spot-check --sid 1484
 ```
diff --git a/docs/projects/spot-latest.md b/docs/projects/spot-latest.md
@@ -0,0 +1,17 @@
+# Global latest Snapshot
+
+This tool allows the user to create the latest data snapshot of global spotter database. An extra column with system:time_start is added where the timestamp is converted to epoch time in milliseconds. This tool will accept dtype/data type which can be wave/wind.
+
+![pyspotter_snapshot_latest](https://github.com/open-oceans/pyspotter/assets/6677629/8a447f60-9f73-42fb-8234-aa2e2902f37d)
+
+```
+pyspotter snapshot-latest -h
+usage: pyspotter snapshot-latest [-h] --export EXPORT --dtype DTYPE
+
+options:
+  -h, --help       show this help message and exit
+
+Required named arguments.:
+  --export EXPORT  Full path to folder to export results
+  --dtype DTYPE    Data type wind/wave
+```
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -139,5 +139,5 @@ nav:
       - Data export: projects/device-export.md
   - Global Tools:
       - Global biweekly Snapshot: projects/snapshot.md
-      - Global latest Snapshot: projects/spotter-latest.md
+      - Global latest Snapshot: projects/spot-latest.md
   - Changelog: changelog.md
diff --git a/pyspotter/pyspotter.py b/pyspotter/pyspotter.py
@@ -74,14 +74,15 @@ def compareVersion(self, version1, version2):
 
 
 # Get package version
+def version_latest(package):
+    response = requests.get(f"https://pypi.org/pypi/{package}/json")
+    latest_version = response.json()["info"]["version"]
+    return latest_version
+
+
 def pyspotter_version():
-    url = "https://pypi.org/project/pyspotter/"
-    source = requests.get(url)
-    html_content = source.text
-    soup = BeautifulSoup(html_content, "html.parser")
-    company = soup.find("h1")
     vcheck = ob1.compareVersion(
-        company.string.strip().split(" ")[-1],
+        version_latest("pyspotter"),
         pkg_resources.get_distribution("pyspotter").version,
     )
     if vcheck == 1:
@@ -92,7 +93,7 @@ def pyspotter_version():
         print(
             "Current version of pyspotter is {} upgrade to lastest version: {}".format(
                 pkg_resources.get_distribution("pyspotter").version,
-                company.string.strip().split(" ")[-1],
+                version_latest("pyspotter"),
             )
         )
         print(
@@ -106,7 +107,7 @@ def pyspotter_version():
         print(
             "Possibly running staging code {} compared to pypi release {}".format(
                 pkg_resources.get_distribution("pyspotter").version,
-                company.string.strip().split(" ")[-1],
+                version_latest("pyspotter"),
             )
         )
         print(
@@ -116,6 +117,19 @@ def pyspotter_version():
 
 pyspotter_version()
 
+# Go to the readMe
+def readme():
+    try:
+        a = webbrowser.open("https://pyspotter.openoceans.xyz", new=2)
+        if a == False:
+            print("Your setup does not have a monitor to display the webpage")
+            print(" Go to {}".format("https://pyspotter.openoceans.xyz"))
+    except Exception as error:
+        print(error)
+
+def read_from_parser(args):
+    readme()
+
 
 # set credentials
 def auth(usr):
@@ -137,6 +151,9 @@ def auth(usr):
     if usr is None:
         usr = input("Enter email: ")
     pwd = getpass.getpass("Enter password: ")
+    while len(pwd) == 0:
+        logging.error("Password cannot be empty")
+        pwd = getpass.getpass("Enter password: ")
     data = {"username": usr, "password": pwd, "skipRedirect": "true"}
 
     response = requests.post(
@@ -448,17 +465,16 @@ def spot_check(spot_id):
             f"Spot check failed with error code {response.status_code}: {response.json()['message']}"
         )
 
-
 def spotcheck_from_parser(args):
     spot_check(spot_id=args.sid)
 
 
-def spot_data(spot_id, dtype, folder):  #'SPOT-0222'
-    waves_list = []
-    wind_list = []
-    sst_list = []
+def spot_data(spot_id, dtype, folder):
+    if dtype == "sst":
+        dtype = "surfaceTemp"
     if not spot_id.startswith("SPOT-"):
         spot_id = f"SPOT-{spot_id}"
+
     obj = TimezoneFinder()
     params = {
         "spotterId": [spot_id],
@@ -468,113 +484,44 @@ def spot_data(spot_id, dtype, folder):  #'SPOT-0222'
     headers = {
         "token": tokenize(),
     }
+
     response = requests.get(
         "https://api.sofarocean.com/api/wave-data", headers=headers, params=params
     )
+
     if response.status_code == 200:
         spotter = response.json()
         print("\n" + f"Fetching info for Spotter {spot_id}" + "\n")
-        if (
-            not "surfaceTemp" in spotter["data"]
-            or len(spotter["data"]["surfaceTemp"]) == 0
-            and dtype == "sst"
-        ):
-            sys.exit("No surfaceTemp data found")
-        else:
-            for readings in spotter["data"]["surfaceTemp"]:
-                readings["date"] = readings["timestamp"].split("T")[0]
-                readings["spotter_id"] = spot_id
-                sst_list.append(readings)
-        if (
-            not "waves" in spotter["data"]
-            or len(spotter["data"]["waves"]) == 0
-            and dtype == "wave"
-        ):
-            sys.exit("No waves data found")
-        else:
-            for readings in spotter["data"]["waves"]:
-                readings["date"] = readings["timestamp"].split("T")[0]
-                readings["spotter_id"] = spot_id
-                waves_list.append(readings)
-        if (
-            not "wind" in spotter["data"]
-            or len(spotter["data"]["wind"]) == 0
-            and dtype == "wind"
-        ):
-            sys.exit("No wind data found")
-        else:
-            for readings in spotter["data"]["wind"]:
-                readings["date"] = readings["timestamp"].split("T")[0]
-                readings["spotter_id"] = spot_id
-                wind_list.append(readings)
+        #print(spotter["data"].get("surfaceTemp"))
+        if not spotter["data"].get(dtype):
+            sys.exit(f"No {dtype} data found")
+
+        data_list = spotter["data"][dtype]
+
+        for readings in data_list:
+            readings["date"] = readings["timestamp"].split("T")[0]
+            readings["spotter_id"] = spot_id
+
+        df = pd.DataFrame(data_list)
+
+        if "date" in df.columns:
+            df.sort_values(by="date", inplace=True)
+            df["timestamp"] = pd.to_datetime(df["timestamp"])
+            df["system:time_start"] = df["timestamp"].apply(
+                datetime_to_epoch_milliseconds
+            )
+            for key, group in df.groupby("date"):
+                print(f"Processing {spot_id}_{key}_{dtype}.csv")
+                group.to_csv(
+                    os.path.join(folder, f"{spot_id}_{key}_{dtype}.csv"),
+                    index=False,
+                    sep=",",
+                )
     else:
         sys.exit(
             f"Failed with status_code: {response.status_code}: {response.json()['message']}"
         )
 
-    if dtype == "wave":
-        csv_columns = [
-            "significantWaveHeight",
-            "peakPeriod",
-            "meanPeriod",
-            "peakDirection",
-            "peakDirectionalSpread",
-            "meanDirection",
-            "meanDirectionalSpread",
-            "timestamp",
-            "latitude",
-            "longitude",
-            "date",
-            "spotter_id",
-        ]
-        main_list = waves_list
-    elif dtype == "wind":
-        csv_columns = [
-            "speed",
-            "direction",
-            "seasurfaceId",
-            "latitude",
-            "longitude",
-            "timestamp",
-            "date",
-            "spotter_id",
-        ]
-        main_list = wind_list
-    elif dtype == "sst":
-        csv_columns = [
-            "degrees",
-            "latitude",
-            "longitude",
-            "timestamp",
-            "date",
-            "spotter_id",
-        ]
-        main_list = sst_list
-
-    # define a fuction for key
-    def key_func(k):
-        return k["date"]
-
-    # sort INFO data by 'company' key.
-    INFO = sorted(main_list, key=key_func)
-
-    for key, value in groupby(INFO, key_func):
-        print(f"Processing {spot_id}_{key}_{dtype}.csv")
-        dict_data = list(value)
-        try:
-            with open(
-                os.path.join(folder, f"{spot_id}_{key}_{dtype}.csv"), "w"
-            ) as csvfile:
-                writer = csv.DictWriter(
-                    csvfile, fieldnames=csv_columns, delimiter=",", lineterminator="\n"
-                )
-                writer.writeheader()
-                for data in dict_data:
-                    writer.writerow(data)
-        except IOError:
-            print("I/O error")
-
-
 def spot_data_from_parser(args):
     spot_data(spot_id=args.sid, dtype=args.dtype, folder=args.folder)
 
@@ -583,6 +530,11 @@ def main(args=None):
     parser = argparse.ArgumentParser(description="Simple CLI for Sofarocean API")
     subparsers = parser.add_subparsers()
 
+    parser_read = subparsers.add_parser(
+        "readme", help="Go to the web based pyspotter cli readme page"
+    )
+    parser_read.set_defaults(func=read_from_parser)
+
     parser_auth = subparsers.add_parser(
         "auth", help="Authenticates and saves your API token"
     )