[moriartym@login02 ~]$ cd saved_fisdat/ [moriartym@login02 saved_fisdat]$ git pull https://github.com/wwaites/saved_fisdat.git remote: Enumerating objects: 155, done. remote: Counting objects: 100% (13/13), done. remote: Compressing objects: 100% (4/4), done. remote: Total 155 (delta 9), reused 10 (delta 9), pack-reused 142 Receiving objects: 100% (155/155), 79.74 KiB | 0 bytes/s, done. Resolving deltas: 100% (94/94), completed with 3 local objects. From https://github.com/wwaites/saved_fisdat * branch HEAD -> FETCH_HEAD Updating f8a9a72..f000e0c Fast-forward .gitmodules | 3 + MANIFEST.in | 2 + README.md | 50 +- examples/farm_site_af_source/fo_farms.yaml | 54 ++ examples/farm_site_af_source/fo_lice.yaml | 53 ++ examples/sentinel_cages/prep.R | 18 + examples/sentinel_cages/sentinel_cages_cleaned.csv | 3892 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ examples/sentinel_cages/sentinel_cages_sampling.yaml | 264 ++++++++++ examples/sentinel_cages/sentinel_cages_site.yaml | 151 ++++++ fisdat/cmd_dat.py | 394 ++++++++++----- fisdat/cmd_up.py | 210 ++++++-- fisdat/data_model | 1 + fisdat/utils.py | 75 +++ setup.py | 8 +- 14 files changed, 4975 insertions(+), 200 deletions(-) create mode 100644 .gitmodules create mode 100644 examples/farm_site_af_source/fo_farms.yaml create mode 100644 examples/farm_site_af_source/fo_lice.yaml create mode 100644 examples/sentinel_cages/prep.R create mode 100644 examples/sentinel_cages/sentinel_cages_cleaned.csv create mode 100644 examples/sentinel_cages/sentinel_cages_sampling.yaml create mode 100644 examples/sentinel_cages/sentinel_cages_site.yaml create mode 160000 fisdat/data_model [moriartym@login02 saved_fisdat]$ module load conda (base) [moriartym@login02 saved_fisdat]$ conda activate fisdat (fisdat) [moriartym@login02 saved_fisdat]$ cd /opt/tmp/SeaLice_Fisdat/ (fisdat) [moriartym@login02 SeaLice_Fisdat]$ ls data_hourly_a11.csv data_hourly_s11.csv manifest.json marinescot.key Schema_long_hourly_model_data.json (fisdat) [moriartym@login02 SeaLice_Fisdat]$ export GOOGLE_APPLICATION_CREDENTIALS=/opt/tmp/SeaLice_Fisdat/marinescot.key (fisdat) [moriartym@login02 SeaLice_Fisdat]$ fisup manifest.json Checking data_hourly_a11.csv ... Checking data_hourly_s11.csv ... Uploading gs://saved-fisdat/marinescot/20240402/0b0b120e-f0c5-11ee-a221-a2bc2da53d52/manifest.json ... Uploading gs://saved-fisdat/marinescot/20240402/0b0b120e-f0c5-11ee-a221-a2bc2da53d52/data_hourly_a11.csv ... Traceback (most recent call last): File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connection.py", line 402, in request self.send(chunk) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/http/client.py", line 1001, in send self.sock.sendall(data) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/ssl.py", line 1205, in sendall v = self.send(byte_view[count:]) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/ssl.py", line 1174, in send return self._sslobj.write(data) socket.timeout: The write operation timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/requests/adapters.py", line 486, in send resp = conn.urlopen( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connectionpool.py", line 844, in urlopen retries = retries.increment( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment raise reraise(type(error), error, _stacktrace) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/util/util.py", line 38, in reraise raise value.with_traceback(tb) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connectionpool.py", line 790, in urlopen response = self._make_request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request conn.request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/urllib3/connection.py", line 402, in request self.send(chunk) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/http/client.py", line 1001, in send self.sock.sendall(data) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/ssl.py", line 1205, in sendall v = self.send(byte_view[count:]) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/ssl.py", line 1174, in send return self._sslobj.write(data) urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/fisdat-0.1-py3.9.egg/fisdat/cmd_up.py", line 35, in upload_files File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_cloud_storage-2.15.0-py3.9.egg/google/cloud/storage/fileio.py", line 357, in write self._upload_chunks_from_buffer(num_chunks) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_cloud_storage-2.15.0-py3.9.egg/google/cloud/storage/fileio.py", line 417, in _upload_chunks_from_buffer upload.transmit_next_chunk(transport, **kwargs) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/requests/upload.py", line 515, in transmit_next_chunk return _request_helpers.wait_and_retry( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/requests/_request_helpers.py", line 178, in wait_and_retry raise error File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/requests/_request_helpers.py", line 155, in wait_and_retry response = func() File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/requests/upload.py", line 507, in retriable_request result = transport.request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_auth-2.28.1-py3.9.egg/google/auth/transport/requests.py", line 541, in request response = super(AuthorizedSession, self).request( File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/requests/adapters.py", line 501, in send raise ConnectionError(err, request=request) requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/moriartym/.conda/envs/fisdat/bin/fisup", line 33, in sys.exit(load_entry_point('fisdat==0.1', 'console_scripts', 'fisup')()) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/fisdat-0.1-py3.9.egg/fisdat/cmd_up.py", line 96, in cli File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/fisdat-0.1-py3.9.egg/fisdat/cmd_up.py", line 35, in upload_files File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_cloud_storage-2.15.0-py3.9.egg/google/cloud/storage/fileio.py", line 437, in close self._upload_chunks_from_buffer(1) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_cloud_storage-2.15.0-py3.9.egg/google/cloud/storage/fileio.py", line 417, in _upload_chunks_from_buffer upload.transmit_next_chunk(transport, **kwargs) File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/requests/upload.py", line 503, in transmit_next_chunk method, url, payload, headers = self._prepare_request() File "/home/moriartym/.conda/envs/fisdat/lib/python3.9/site-packages/google_resumable_media-2.7.0-py3.9.egg/google/resumable_media/_upload.py", line 627, in _prepare_request raise ValueError(msg) ValueError: Bytes stream is in unexpected state. The local stream has had 41943040 bytes read from it while 0 bytes have already been updated (they should match). (fisdat) [moriartym@login02 SeaLice_Fisdat]$