Error using HSDS to download NSRDB data

[acheyette]

I have set up a local HSDS server using a version of these steps. I am trying to run SAM using nsrdb data with the get_SAM_gid function (i.e., a small sample of my code looks like

with NSRDBX(f'/nrel/nsrdb/current/nsrdb_{year}.h5', hsds=True) as f:
        data_segment = f.get_SAM_gid(gid_set)

where gid_set is a set of gids in the NSRDB data. However, recently when I have been trying to run this code, I am getting a connection error with the HSDS server. This is a snippet of the output from the terminal where I start the HSDS server:

(hsds) annacheyette@b0-be-83-57-dd-ce hsds % sh runall.sh --no-docker-tcp
use tcp with no-docker option
using password file: admin/config/passwd.default
AWS_S3_GATEWAY set, using nrel-pds-hsds S3 Bucket (verify that this bucket exists)
no docker startup
Using S3 Gateway
set logging to:: 20
logfile: /Users/annacheyette/hsds/hs.log
got command line arg for config_dir: admin/config
INFO:root:using cmd_dir: /opt/homebrew/bin
got command line arg for config_dir: admin/config
got command line arg for config_dir: admin/config
got command line arg for config_dir: admin/config
got command line arg for config_dir: admin/config
got command line arg for config_dir: admin/config
INFO:root:all processes ready!
INFO:root:Ready after: 5.03 s

READY! use endpoint: http://localhost:5101

Exception in callback RequestHandler.connection_made(<_SelectorSoc...e, bufsize=0>>)
handle: <Handle RequestHandler.connection_made(<_SelectorSoc...e, bufsize=0>>)>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 275, in connection_made
    tcp_keepalive(real_transport)
  File "/opt/homebrew/lib/python3.11/site-packages/aiohttp/tcp_helpers.py", line 16, in tcp_keepalive
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
  File "/opt/homebrew/Cellar/python@3.11/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/asyncio/trsock.py", line 74, in setsockopt
    self._sock.setsockopt(*args, **kwargs)
OSError: [Errno 22] Invalid argument
Exception in callback RequestHandler.connection_made(<_SelectorSoc...e, bufsize=0>>)
handle: <Handle RequestHandler.connection_made(<_SelectorSoc...e, bufsize=0>>)>

and here is a snippet from the hs.log file:

sn INFO> read_chunk_hyperslab, chunk_id: c-02780ea8-a9348af3-ddd3-ed384f-f74652_14, bucket: nrel-pds-hsds
dn2 WARN> expected to find c-02780ea8-a9348af3-ddd3-ed384f-f74652_5 in pending_s3_read map
dn3 WARN> expected to find c-02780ea8-a9348af3-ddd3-ed384f-f74652_9 in pending_s3_read map
dn4 INFO> s3Client.get_object(current/nsrdb_2021.h5[86539472:90639152] bucket=nrel-pds-nsrdb) start=1702587810.5875 finish=1702587874.0361 elapsed=63.4486 bytes=4099680
sn INFO> http_get('http://localhost:6102/chunks/c-02780ea8-a9348af3-ddd3-ed384f-f74652_14')
dn2 ERROR> Exception during binary data write: Cannot write to closing transport
dn3 ERROR> Exception during binary data write: Cannot write to closing transport
dn4 INFO> read: 4099680 bytes for key: current/nsrdb_2021.h5
sn INFO> http_get status: 200 for req: http://localhost:6102/chunks/c-02780ea8-a9348af3-ddd3-ed384f-f74652_14
dn2 INFO> s3Client.get_object(current/nsrdb_2021.h5[66041072:70140752] bucket=nrel-pds-nsrdb) start=1702587811.0324 finish=1702587873.8726 elapsed=62.8402 bytes=4099680
dn3 INFO> s3sync nothing to update
dn4 WARN> expected to find c-02780ea8-a9348af3-ddd3-ed384f-f74652_21 in pending_s3_read map
sn INFO> chunk_arr shape: (31536,)
dn2 INFO> read: 4099680 bytes for key: current/nsrdb_2021.h5
dn3 INFO> s3syncCheck no objects to write, sleeping for 1.00
dn4 ERROR> Exception during binary data write: Cannot write to closing transport
sn INFO> data_sel: (slice(441504, 473040, 1),)
dn2 WARN> expected to find c-02780ea8-a9348af3-ddd3-ed384f-f74652_16 in pending_s3_read map
dn4 INFO> s3Client.get_object(current/nsrdb_2021.h5[61941392:66041072] bucket=nrel-pds-nsrdb) start=1702587780.8125 finish=1702587874.2291 elapsed=93.4166 bytes=4099680
sn INFO> np_arr shape: (672755,)
dn2 ERROR> Exception during binary data write: Cannot write to closing transport
dn4 INFO> read: 4099680 bytes for key: current/nsrdb_2021.h5
sn INFO> ChunkCrawler - worker status for chunk c-02780ea8-a9348af3-ddd3-ed384f-f74652_14: 200
dn2 REQ> GET: /chunks/c-02780ea8-a9348af3-ddd3-ed384f-f74652_14 [localhost:6102]
dn4 WARN> expected to find c-02780ea8-a9348af3-ddd3-ed384f-f74652_15 in pending_s3_read map
sn INFO> ChunkCrawler - join complete - count: 22
dn2 INFO> get_metadata_obj: d-02780ea8-a9348af3-ddd3-ed384f-f74652 bucket: nrel-pds-hsds
dn4 ERROR> Exception during binary data write: Cannot write to closing transport
sn INFO> doReadSelection complete - status:  200
dn2 REQ> GET: /chunks/c-02780ea8-a9348af3-ddd3-ed384f-f74652_14 [localhost:6102]
dn4 INFO> s3Client.get_object(current/nsrdb_2021.h5[61941392:66041072] bucket=nrel-pds-nsrdb) 

It appears there is some sort of connection error with the server where the nsrdb server is stored. Do you have any ideas about what could be causing this issue? Thank you!