updated v0.4.7

- Download selenium drivers for all operating systems - updated asset size, acl changer and other tools - fixed metadata tool to work with python3 - overall general improvements to the tool and updates - added selenium setup tools
samapriya · Oct 5, 2019 · 474eafa · 474eafa
1 parent bdcaff7
commit 474eafa
Show file tree

Hide file tree

Showing 25 changed files with 3,935 additions and 2,261 deletions.
diff --git a/README.md b/README.md
@@ -8,8 +8,7 @@
 
 This tool is designed to facilitate moving data from Planet's API into Google Earth Engine and using a metadata library. The tool downloads data into a local storage and allows you to process the metadata before uploading into Google Earth Engine. This tool has been updated to account for metadata property type by going through each metadata column and then uses the **upload manifest** function to upload images for ingestion to EE. This tool also draws from an additional tool I created with is the [Google Earth Engine Asset Manager Addon](https://github.com/samapriya/gee_asset_manager_addon) This includes the batch upload feature, but now additional tools such as generating reports of Earth Engine assets and querying quota to name just a few. The ambition is apart from helping users with batch actions on assets along with interacting and extending capabilities of existing GEE CLI. It is developed case by case basis to include more features in the future as it becomes available or as the need arises. I have now released this as a [PyPI package](https://pypi.org/project/ppipe/) for easy installation and this will be updated along with the the GitHub package.
 
-![ppipe_main](https://user-images.githubusercontent.com/6677629/57460065-6f2ae180-7242-11e9-864a-fd327edee950.png)
-
+Though this tool is designed to download planet imagery, use [porder](https://github.com/samapriya/porder) to use the new ordersv2 API to download planet imagery. This includes additional filter capabilities along with capability to clip and download.
 
 ## Table of contents
 * [Installation](#installation)
@@ -23,6 +22,7 @@ This tool is designed to facilitate moving data from Planet's API into Google Ea
 	* [Metadata Parser](#metadata-parser)
 * [Earth Engine Tools](#earth-engine-tools)
     * [selenium update](#selenium-update)
+    * [gee selsetup](#gee-selsetup)
 	* [EE User](#ee-user)
 	* [EE Quota](#ee-quota)
 	* [Create](#create)
@@ -77,15 +77,16 @@ Installation is an optional step; the application can be also run directly by ex
 As usual, to print help:
 
 ```
+> ppipe -h
 usage: ppipe [-h]
              {
-             ,planetkey,pquota,dasync,savedsearch,metadata,update,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
+             ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,ca
              ...
 
 Planet Pipeline with Google Earth Engine Batch Addons
 
 positional arguments:
-  { ,planetkey,pquota,dasync,savedsearch,metadata,update,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
+  { ,planetkey,pquota,dasync,savedsearch,metadata,update,selsetup,ee_user,quota,create,selupload,lst,assetsize,tasks,access,cancel}
                         ---------------------------------------
                         -----Choose from Planet Tools Below-----
                         ---------------------------------------
@@ -99,8 +100,9 @@ positional arguments:
                         -------------------------------------------
                         ----Choose from Earth Engine Tools Below----
                         -------------------------------------------
-    update              Updates Selenium drivers for firefox [windows or linux
-                        systems]
+    update              Updates Selenium drivers for firefox
+    selsetup            Non headless setup for new google account, use if
+                        upload throws errors
     ee_user             Get Earth Engine API Key & Paste it back to Command
                         line/shell to change user
     quota               Print Earth Engine total quota and used quota
@@ -247,6 +249,11 @@ The ambition is apart from helping user with batch actions on assets along with
 
 ``` ppipe update```
 
+### gee selsetup
+Once in a while the geckodriver requires manual input before signing into the google earth engine, this tool will allow you to interact with the initialization of Google Earth Engine code editor window. It allows the user to specify the account they want to use, and should only be needed once.
+
+```geeup selsetup```
+
 ### EE User
 This tool is designed to allow different users to change earth engine authentication credentials. The tool invokes the authentication call and copies the authentication key verification website to the clipboard which can then be pasted onto a browser and the generated key can be pasted back. This command takes in no arguments.
 
@@ -380,6 +387,13 @@ Original upload function adapted from [Lukasz's asset manager tool](https://gith
 
 # Changelog
 
+### v0.4.7
+- Download selenium drivers for all operating systems
+- updated asset size, acl changer and other tools
+- fixed metadata tool to work with python3
+- overall general improvements to the tool and updates
+- added selenium setup tools
+
 ### v0.4.6
 - added fix for downloading selenium drivers
 - improved error handling

diff --git a/ppipe/__init__.py b/ppipe/__init__.py
@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
 
-__author__ = 'Samapriya Roy'
-__email__ = 'samapriya.roy@gmail.com'
-__version__ = '0.4.6'
+__author__ = "Samapriya Roy"
+__email__ = "samapriya.roy@gmail.com"
+__version__ = "0.4.7"
diff --git a/ppipe/acl_changer.py b/ppipe/acl_changer.py
@@ -1,4 +1,5 @@
 from __future__ import print_function
+
 __copyright__ = """
 
     Copyright 2019 Samapriya Roy
@@ -18,6 +19,7 @@
 """
 __license__ = "Apache 2.0"
 
+
 import ee
 import os
 import json
@@ -26,64 +28,64 @@
 ee.Initialize()
 
 ##request type of asset, asset path and user to give permission
-def access(collection_path,user,role):
-    header=ee.data.getInfo(collection_path)['type']
-    if header=="ImageCollection":
-        lst=[]
+def access(collection_path, user, role):
+    header = ee.data.getInfo(collection_path)["type"]
+    if header == "ImageCollection":
+        lst = []
         lst.append(collection_path)
-        assets_names=lst
-    elif header=="Image":
-        lst=[]
+        assets_names = lst
+    elif header == "Image":
+        lst = []
         lst.append(collection_path)
-        assets_names=lst
-    elif header=="Table":
-        lst=[]
+        assets_names = lst
+    elif header == "Table":
+        lst = []
         lst.append(collection_path)
-        assets_names=lst
+        assets_names = lst
     else:
-        assets_list = ee.data.getList(params={'id': collection_path})
-        assets_names = [os.path.basename(asset['id']) for asset in assets_list]
-    print('Changing permission for total of '+str(len(assets_names))+'.....')
-    for count,items in enumerate(assets_names):
-        if header=='Folder':
-            init=collection_path+'/'+items
-            print('Working on item '+str(init))
+        assets_list = ee.data.getList(params={"id": collection_path})
+        assets_names = [os.path.basename(asset["id"]) for asset in assets_list]
+    print("Changing permission for total of " + str(len(assets_names)) + ".....")
+    for count, items in enumerate(assets_names):
+        if header == "Folder":
+            init = collection_path + "/" + items
+            print("Working on item " + str(init))
         else:
-            init=items
-            print('Working on item '+str(init))
+            init = items
+            print("Working on item " + str(init))
         acl = ee.data.getAssetAcl(init)
-        if role=='reader':
-            if not user in acl['readers']:
-                baselist=acl['readers']
+        if role == "reader":
+            if not user in acl["readers"]:
+                baselist = acl["readers"]
                 baselist.append(user)
-                acl['readers']=baselist
-                acl['owners']=[]
+                acl["readers"] = baselist
+                acl["owners"] = []
                 try:
                     ee.data.setAssetAcl(init, json.dumps(acl))
                 except Exception as e:
                     print(e)
             else:
-                print('user already has read access to this asset:SKIPPING')
-        if role=='writer':
-            if not user in acl['writers']:
-                baselist=acl['writers']
+                print("user already has read access to this asset:SKIPPING")
+        if role == "writer":
+            if not user in acl["writers"]:
+                baselist = acl["writers"]
                 baselist.append(user)
-                acl['readers']=baselist
-                acl['owners']=[]
+                acl["readers"] = baselist
+                acl["owners"] = []
                 try:
                     ee.data.setAssetAcl(init, json.dumps(acl))
                 except Exception as e:
                     print(e)
             else:
-                print('user already has write access to this asset:SKIPPING')
-        if role=='delete':
-            if not user in acl['readers']:
-                print('user does not have permission:SKIPPING')
+                print("user already has write access to this asset:SKIPPING")
+        if role == "delete":
+            if not user in acl["readers"]:
+                print("user does not have permission:SKIPPING")
             else:
-                baselist=acl['readers']
+                baselist = acl["readers"]
                 baselist.remove(user)
-                acl['readers']=baselist
-                acl['owners']=[]
+                acl["readers"] = baselist
+                acl["owners"] = []
                 try:
                     ee.data.setAssetAcl(init, json.dumps(acl))
                 except Exception as e:

diff --git a/ppipe/assetsizes.py b/ppipe/assetsizes.py
@@ -1,4 +1,5 @@
 from __future__ import print_function
+
 __copyright__ = """
 
     Copyright 2019 Samapriya Roy
@@ -18,41 +19,62 @@
 """
 __license__ = "Apache 2.0"
 
-import ee,subprocess,os
+import ee, subprocess, os
+
 ##initialize earth engine
 ee.Initialize()
 
-suffixes = ['B', 'KB', 'MB', 'GB', 'TB', 'PB']
+suffixes = ["B", "KB", "MB", "GB", "TB", "PB"]
+
+
 def humansize(nbytes):
     i = 0
-    while nbytes >= 1024 and i < len(suffixes)-1:
-        nbytes /= 1024.
+    while nbytes >= 1024 and i < len(suffixes) - 1:
+        nbytes /= 1024.0
         i += 1
-    f = ('%.2f' % nbytes).rstrip('0').rstrip('.')
-    return '%s %s' % (f, suffixes[i])
-l=[]
+    f = ("%.2f" % nbytes).rstrip("0").rstrip(".")
+    return "%s %s" % (f, suffixes[i])
+
+
+l = []
+
+
 def assetsize(asset):
-    header=ee.data.getInfo(asset)['type']
-    if header=="ImageCollection":
-        collc=ee.ImageCollection(asset)
-        size=collc.aggregate_array('system:asset_size')
-        print('')
-        print(str(asset)+" ===> "+str(humansize(sum(size.getInfo()))))
-        print('Total number of items in collection: '+str(collc.size().getInfo()))
-    elif header=="Image":
-        collc=ee.Image(asset)
-        print('')
-        print(str(asset)+" ===> "+str(humansize(collc.get('system:asset_size').getInfo())))
-    elif header=="Table":
-        collc=ee.FeatureCollection(asset)
-        print('')
-        print(str(asset)+" ===> "+str(humansize(collc.get('system:asset_size').getInfo())))
-    elif header =="Folder":
-        b=subprocess.check_output("earthengine du "+asset+" -s",shell=True)
-        num=subprocess.check_output("earthengine ls "+asset,shell=True)
-        size=humansize(float(b.strip().split(' ')[0]))
-        print('')
-        print(str(asset)+" ===> "+str(size))
-        print('Total number of items in folder: '+str(len(num.split('\n'))-1))
-if __name__ == '__main__':
+    header = ee.data.getInfo(asset)["type"]
+    if header == "ImageCollection":
+        collc = ee.ImageCollection(asset)
+        size = collc.aggregate_array("system:asset_size")
+        print("")
+        print(str(asset) + " ===> " + str(humansize(sum(size.getInfo()))))
+        print("Total number of items in collection: " + str(collc.size().getInfo()))
+    elif header == "Image":
+        collc = ee.Image(asset)
+        print("")
+        print(
+            str(asset)
+            + " ===> "
+            + str(humansize(collc.get("system:asset_size").getInfo()))
+        )
+    elif header == "Table":
+        collc = ee.FeatureCollection(asset)
+        print("")
+        print(
+            str(asset)
+            + " ===> "
+            + str(humansize(collc.get("system:asset_size").getInfo()))
+        )
+    elif header == "Folder":
+        b = subprocess.check_output(
+            "earthengine --no-use_cloud_api du " + asset + " -s", shell=True
+        )
+        num = subprocess.check_output(
+            "earthengine --no-use_cloud_api ls " + asset, shell=True
+        )
+        size = humansize(float(b.strip().split(" ")[0]))
+        print("")
+        print(str(asset) + " ===> " + str(size))
+        print("Total number of items in folder: " + str(len(num.split("\n")) - 1))
+
+
+if __name__ == "__main__":
     assetsize(None)