Various bugs in geeup

[adamjstewart]

Hi, I'm a first-time geeup user. I just wanted to report my experience so far and see if you can fix any of the bugs I encountered.

After installing geeup from source, I ran:

$ geeup update
Downloading from: https://github.com/mozilla/geckodriver/releases/download/v0.24.0/geckodriver-v0.24.0-linux64.tar.gz
 [*] 2.8 MB / 2.8 MB @ 2.2 MB/s [##################] [100%, 0s left]     
/mnt/a/u/sciteam/stewart1/spack/opt/spack/cray-cnl5-interlagos/gcc-6.3.0/py-geeup-0.2.4-ars3milv5bauueoa2k25vac45bte44qo/lib/python3.7/site-packages/geeup/geckodriver-v0.24.0-linux64.zip
Use selenium driver path as /mnt/a/u/sciteam/stewart1/spack/opt/spack/cray-cnl5-interlagos/gcc-6.3.0/py-geeup-0.2.4-ars3milv5bauueoa2k25vac45bte44qo/lib/python3.7/site-packages/geeup/geckodriver

First, I tried to zip some shapefiles:

$ geeup zipshape --input 31/001 --output 31/001

Although there was no error message reported, the .shp file and friends located in 31/001 were not zipped. No zipfile was created.

Next I tried uploading the tables:

$ geeup seltabup --source 31/001 --dest /users/ajstewart/hummingbird/31001 --email ajstewart426@gmail.com
Traceback (most recent call last):
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/bin/geeup", line 12, in <module>
    load_entry_point('geeup==0.2.4', 'console_scripts', 'geeup')()
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 149, in main
    args.func(args)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 85, in seltabup_from_parser
    destination=args.dest)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 60, in seltabup
    create_image_collection(destination)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 48, in create_image_collection
    if folder_exist(full_path_to_collection):
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 41, in folder_exist
    if ee.data.getInfo(path) and ee.data.getInfo(path)['type']=='Folder':
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/ee/deprecation.py", line 32, in Wrapper
    return func(*args, **kwargs)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/ee/data.py", line 385, in getInfo
    return send_('/info', {'id': asset_id})
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/ee/data.py", line 1717, in send_
    raise ee_exception.EEException(json_content['error']['message'])
ee.ee_exception.EEException: Path for the asset is invalid. Folder names must be separated with '/', and at most 10 folders are allowed in the path. Names must contain only the following characters: A..Z, a..z, 0..9, '_' or '-'. Maximum length for a single folder or asset name is 100 characters.

Although the error message isn't very helpful, the bug was that I erroneously placed a / at the start of my dest. Next I tried removing that:

$ geeup seltabup --source 31/001 --dest users/ajstewart/hummingbird/31001 --email ajstewart426@gmail.com
Traceback (most recent call last):
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/bin/geeup", line 12, in <module>
    load_entry_point('geeup==0.2.4', 'console_scripts', 'geeup')()
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 149, in main
    args.func(args)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 85, in seltabup_from_parser
    destination=args.dest)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 60, in seltabup
    create_image_collection(destination)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 51, in create_image_collection
    ee.data.createAsset({'type': ee.data.ASSET_TYPE_FOLDER}, full_path_to_collection)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/ee/data.py", line 932, in createAsset
    return send_('/create', args)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/ee/data.py", line 1717, in send_
    raise ee_exception.EEException(json_content['error']['message'])
ee.ee_exception.EEException: Asset 'users/ajstewart/hummingbird' does not exist or doesn't allow this operation.

Although geeup claims to be able to automatically create any parent directories, this does not appear to be the case. After manually creating the hummingbird directory in GEE, I get:

$ geeup seltabup --source 31/001 --dest users/ajstewart/hummingbird/31001 --email ajstewart426@gmail.com
Folder users/ajstewart/hummingbird already exists
Traceback (most recent call last):
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/bin/geeup", line 12, in <module>
    load_entry_point('geeup==0.2.4', 'console_scripts', 'geeup')()
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 149, in main
    args.func(args)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/geeup.py", line 85, in seltabup_from_parser
    destination=args.dest)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/geeup/sel_tuploader.py", line 64, in seltabup
    driver = Firefox(executable_path=os.path.join(lp,"geckodriver"),firefox_options=options)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/selenium/webdriver/firefox/webdriver.py", line 174, in __init__
    keep_alive=True)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/mnt/a/u/sciteam/stewart1/crop_classification/.spack-env/view/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: invalid argument: can't kill an exited process

No idea what to do about this one. I'm launching this from a supercomputer, so there's a chance there is something wrong with the Selenium installation. I'm not even sure if Firefox is installed. Overall, more helpful error messages would be great. I'll try running from my laptop, but if that doesn't work, I guess I'll be forced to upload things manually.

[adamjstewart]

When I try to run on my laptop (macOS 10.14.6), I get:

$ geeup update
Architecture not recognized

As before, geeup zipshape does nothing, and I don't expect geeup seltabup to do much without the Selenium drivers. For now, I'll try https://github.com/tracek/gee_asset_manager and hope that it's a little more stable.

[adamjstewart]

Okay, it looks like https://github.com/tracek/gee_asset_manager only supports ImageCollections, not FeatureCollections. And the GEE CLI doesn't support local uploads. Guess I'll manually upload everything...

[samapriya]

Hi @adamjstewart
Selenium only supports environments with a display and not a supercomputing environment, hence the upload tool within geeup will not work in your current setup. Also you are right, currently the tool does not download the selenium drivers for mac os and I will fix that in the next release that should allow you to upload all shapefiles into a folder (not a collection). I am keeping this issue open till I push the updated version to pypi and github.

[adamjstewart]

Thanks! It would be great to find a workaround for supercomputing since that's where I do all of my work, but if I can copy everything to my laptop and upload from there that's almost as good.

[samapriya]

Hi @adamjstewart
Updated version 0.2.5 should now allow you to download selenium drivers for your mac os. Closing for now based on update

[samapriya]

Hi @adamjstewart
I fixed the path issue in the release today so it can get the path for geckodriver correctly for macOS. Upgrade to v0.2.6 before trying this on your mac.

[samapriya]

Hi @adamjstewart
Seems there was an issue with platform for macos being detected as Darwin instead of darwin. It was a minor fix and I updated the release to 0.2.7. You might want to update to this one, if the mac os selenium driver still fails

[clairecw]

I have the latest version installed and am still running into @adamjstewart's issue above, where no zipfile is created at all.