PySAM does not handle 404 responses properly when fetching the weather files #94

ghazi-git · 2021-07-27T14:27:18Z

This issue happened while using pysam to get the weather file for these coordinates (lng=-76.2002669, lat=43.0402873). The problem is that the current implementation expects the response to be always json when the response.status >= 400. However, a 404 response returns some html instead.

It should be noted that I was not able to replicate the error again (running the same code with the above coordinates returns the weather file). Some possible explanations:

I checked the API calls that happen when getting the weather file. A successful response from https://gds-hsds-nsrdb-files.nrelcloud.org usually has the header x-amz-expiration: expiry-date="Thu, 29 Jul 2021 00:00:00 GMT", rule-id="24 hour retention" on it. The error I'm reporting happened at Jul 17, 2021 12:33:53 AM UTC which seems to be somewhat close to when those files are usually set to expire.
some temporary issue happened at the time of the request that made the file unavailable.

Even though the issue can't be replicated, I think it's worth it to fix it.

Additional Info

The API call in question that returned a 404 instead of the expected csv file

GET https://gds-hsds-nsrdb-files.nrelcloud.org/data/solar/1171700_43.05_-76.22_tmy-2020_86eaf692167c07fd77a315d87871e274.csv

PySAM expected the response to be json while the response was actually some html

<html>
<head>
  <title>404 Not Found</title>
</head>
<body>
  <h1>404 Not Found</h1>
  <ul>
    <li>Code: NoSuchKey</li>
    <li>Message: The specified key does not exist.</li>
    <li>Key: data/solar/1171700_43.05_-76.22_tmy-2020_86eaf692167c07fd77a315d87871e274.csv</li>
    <li>RequestId: ...</li>
    <li>HostId: ...</li>
  </ul>
  <hr/>
</body>
</html>

The relevant part of the traceback

  ...
  File "path/to/helpers.py", line 51, in get_weather_file_from_api
    fetcher = wtkfetcher.fetch(lon_lats)
  File "PySAM/ResourceTools.py", line 577, in fetch
    results.append(self.data_function(job))
  File "PySAM/ResourceTools.py", line 503, in _NSRDB_worker
    data_response_json = data_response.json()
  File "requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

The code used to get the weather file

import tempfile

def get_weather_file_from_api(lng, lat) -> str:
    weather_files_tmp_dir = os.path.join(tempfile.gettempdir(), "pysam")
    wtkfetcher = tools.FetchResourceFiles(
        tech="solar",
        nrel_api_key="API_KEY",
        nrel_api_email="EMAIL",
        resource_dir=weather_files_tmp_dir,
    )
    lon_lats = [(lng, lat)]
    fetcher = wtkfetcher.fetch(lon_lats)
    return fetcher.resource_file_paths[0]

The text was updated successfully, but these errors were encountered:

grgmiller · 2021-11-02T22:03:52Z

I believe that I am also experiencing the same issue when downloading files from wind toolkit using pysam. When I run my code a second time, the API call works.

I am getting the following error when running wtkfetcher.fetch(lon_lats):


~\.conda\envs\switch_247\lib\site-packages\PySAM\ResourceTools.py in fetch(self, points)
--> 545                 results.append(self.data_function(job))

~\.conda\envs\switch_247\lib\site-packages\PySAM\ResourceTools.py in _windtk_worker(self, job)
--> 523                 data_response_json = data_response.json()

~\.conda\envs\switch_247\lib\site-packages\requests\models.py in json(self, **kwargs)
--> 910         return complexjson.loads(self.text, **kwargs)

~\.conda\envs\switch_247\lib\json\__init__.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
--> 357         return _default_decoder.decode(s)

~\.conda\envs\switch_247\lib\json\decoder.py in decode(self, s, _w)
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())

~\.conda\envs\switch_247\lib\json\decoder.py in raw_decode(self, s, idx)
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package * Apple Silicon and Intel Wheel pip3 install --extra-index-url https://test.pypi.org/simple/ NREL_PySAM * update files and test * PySAM documentation revisions * update setup and tests * update files * update versions.rst * update rsts * updates for v3.0.2 * update examples Co-authored-by: sjanzou <sjanzou@gmail.com> Co-authored-by: sjanzou <steven@janzouconsulting.com> Co-authored-by: Paul Gilman <cpaulgilman@gmail.com>

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package * Apple Silicon and Intel Wheel pip3 install --extra-index-url https://test.pypi.org/simple/ NREL_PySAM * update files and test * Add utility rate forecast example for new stateful cmod (#119) * Add utility rate forecast example for new stateful cmod * Update failing test_Reopt_sizing.py Add sys.SolarResource.wf_use_albedo = 0 * some stubs changes from develop. Additional documentation for utility rate forecast script * Additional subs files from comd_utilityrateforcast sam branches and develop * Correct energy charge schedule fields to match number of periods in energy charge fields * add tests and improve comments in example Co-authored-by: sjanzou <sjanzou@gmail.com> * PySAM documentation revisions * update setup and tests * update files * update versions.rst * update rsts * updates for v3.0.2 * update build_manylinux.sh * update comments * Modifications to PySSC to handle the conventions form ssc PR 907, plus module and docs updates to match * Pysam 116 urdb version 8 (#126) * Add resourcetools processing for urdbv8, update examples. currently fails on ur_yearzero_usage_peaks * Add new loadtools function to convert hourly data into monthly peaks * Mark urdb version as depricated. Move urdbv8 to new utility rate tools library. Update examples and tests * Fix #128 - correctly apply urdb flatdemandmonths to determine tiers * pysam v4.0.0 * update test * add error check to build_win.bat * fix docs * update cmake version Co-authored-by: sjanzou <sjanzou@gmail.com> Co-authored-by: sjanzou <steven@janzouconsulting.com> Co-authored-by: Brian Mirletz <brian.mirletz@nrel.gov> Co-authored-by: Paul Gilman <cpaulgilman@gmail.com>

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package * Apple Silicon and Intel Wheel pip3 install --extra-index-url https://test.pypi.org/simple/ NREL_PySAM * Latest develop branches of ssc and SAM - maybe after v3.0.2 * Start on shell scripts for arm64 Mac wheel and aarch64 linux wheels * building arm64 python 3.8 on apple silicon * remove .zip files from commit * PySAM 4.0.0 for Apple Silicon * Apple Silicon arm64 and manylinux aarch64 shell files for building wheels Co-authored-by: dguittet <dguittet@nrel.gov>

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package * Apple Silicon and Intel Wheel pip3 install --extra-index-url https://test.pypi.org/simple/ NREL_PySAM * update files and test * Add utility rate forecast example for new stateful cmod (#119) * Add utility rate forecast example for new stateful cmod * Update failing test_Reopt_sizing.py Add sys.SolarResource.wf_use_albedo = 0 * some stubs changes from develop. Additional documentation for utility rate forecast script * Additional subs files from comd_utilityrateforcast sam branches and develop * Correct energy charge schedule fields to match number of periods in energy charge fields * add tests and improve comments in example Co-authored-by: sjanzou <sjanzou@gmail.com> * PySAM documentation revisions * update setup and tests * update files * update versions.rst * update rsts * updates for v3.0.2 * update build_manylinux.sh * update comments * Modifications to PySSC to handle the conventions form ssc PR 907, plus module and docs updates to match * Pysam 116 urdb version 8 (#126) * Add resourcetools processing for urdbv8, update examples. currently fails on ur_yearzero_usage_peaks * Add new loadtools function to convert hourly data into monthly peaks * Mark urdb version as depricated. Move urdbv8 to new utility rate tools library. Update examples and tests * Fix #128 - correctly apply urdb flatdemandmonths to determine tiers * pysam v4.0.0 * update test * add error check to build_win.bat * fix docs * update cmake version * update platform tag for manylinux2014 * update files * Fix wind resource download in ResourceTools.py Now that Windpower supports WIND Toolkit CSV file format, no need to convert to SRW. * update build files --------- Co-authored-by: sjanzou <sjanzou@gmail.com> Co-authored-by: sjanzou <steven@janzouconsulting.com> Co-authored-by: Brian Mirletz <brian.mirletz@nrel.gov> Co-authored-by: Paul Gilman <cpaulgilman@gmail.com>

* files for pysam 2.2.4 * update build_manylinux.sh * update tests and conda build * update SAM_api files * closes #89 certifi certificates in fetchresource * fixes #94 * start v 2.3.0 * fix #69 * changes from 2021.12.2 release * add VersionChanges.rst * add version_differ.py * add more release info * update version_differ.py doc str * correct version_differ.py with PySSC * add python 3.10 * fix getting env variables during build and remove pandas dependency * fix test_ResouceTools.py * switch to manylinux2010 for Python 3.10 support * update pysam files * update PySAM files * Update docs for Version 3.0.1 * close #109 * remove stubs package dependency * removal of stubs package * Apple Silicon and Intel Wheel pip3 install --extra-index-url https://test.pypi.org/simple/ NREL_PySAM * update files and test * Add utility rate forecast example for new stateful cmod (#119) * Add utility rate forecast example for new stateful cmod * Update failing test_Reopt_sizing.py Add sys.SolarResource.wf_use_albedo = 0 * some stubs changes from develop. Additional documentation for utility rate forecast script * Additional subs files from comd_utilityrateforcast sam branches and develop * Correct energy charge schedule fields to match number of periods in energy charge fields * add tests and improve comments in example Co-authored-by: sjanzou <sjanzou@gmail.com> * PySAM documentation revisions * update setup and tests * update files * update versions.rst * update rsts * updates for v3.0.2 * update build_manylinux.sh * update comments * Modifications to PySSC to handle the conventions form ssc PR 907, plus module and docs updates to match * Pysam 116 urdb version 8 (#126) * Add resourcetools processing for urdbv8, update examples. currently fails on ur_yearzero_usage_peaks * Add new loadtools function to convert hourly data into monthly peaks * Mark urdb version as depricated. Move urdbv8 to new utility rate tools library. Update examples and tests * Fix #128 - correctly apply urdb flatdemandmonths to determine tiers * pysam v4.0.0 * update test * add error check to build_win.bat * fix docs * update cmake version * update platform tag for manylinux2014 * update files * Fix wind resource download in ResourceTools.py Now that Windpower supports WIND Toolkit CSV file format, no need to convert to SRW. * Update failing test and build arm64 and aarch64 4.0.1 * twine works for arm64 wheels but manylinux1_aarch64 not supported anaconda upload fails with Extracting conda package attributes for upload Creating package "nrel-pysam" Creating release "4.1.0" [ERROR] ('"nrel-pysam" could not be found', 404) --------- Co-authored-by: dguittet <dguittet@nrel.gov> Co-authored-by: Brian Mirletz <brian.mirletz@nrel.gov> Co-authored-by: Paul Gilman <cpaulgilman@gmail.com>

ghazi-git mentioned this issue Jul 27, 2021

PySAM makes 5 API calls to get the csv weather file #95

Closed

dguittet added a commit that referenced this issue Dec 15, 2021

fixes #94

c188d37

dguittet closed this as completed Dec 15, 2021

dguittet mentioned this issue Jan 5, 2022

PySAM 3.0.0 Release #106

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PySAM does not handle 404 responses properly when fetching the weather files #94

PySAM does not handle 404 responses properly when fetching the weather files #94

ghazi-git commented Jul 27, 2021

grgmiller commented Nov 2, 2021

PySAM does not handle 404 responses properly when fetching the weather files #94

PySAM does not handle 404 responses properly when fetching the weather files #94

Comments

ghazi-git commented Jul 27, 2021

Additional Info

grgmiller commented Nov 2, 2021