Unable to add reader from a local file and import landmask

[mevo-creator]

Hello Team,
First of all, Congratulations and thanks so much for developing easy to use open source trajectory modelling software. I'm new to python and hope i can use this software for my PhD research here at University of Stavanger in Norway. I've installed Opendrift and trying to run tutorials and examples. However, when running tutorials, I'm getting error when trying to add reader from a local file and also when trying to import landmask. Can someone please guide me about this issue. I'm pasting errors below.
Thanks in advance!

-------------Error while trying to add reader from a local file--------------

reader_norkyst = reader_netCDF_CF_generic.Reader('norkyst800_16Nov2015.nc')
13:40:31 INFO: Opening dataset: norkyst800_16Nov2015.nc
13:40:31 INFO: Opening file with Dataset
Traceback (most recent call last):
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift\readers\reader_netCDF_CF_generic.py", line 142, in init
self.Dataset = Dataset(filename, 'r')
File "netCDF4_netCDF4.pyx", line 2321, in netCDF4._netCDF4.Dataset.init
File "netCDF4_netCDF4.pyx", line 1885, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: b'norkyst800_16Nov2015.nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift\readers\reader_netCDF_CF_generic.py", line 144, in init
raise ValueError(e)
ValueError: [Errno 2] No such file or directory: b'norkyst800_16Nov2015.nc'

-------------------- Error when importing landmask--------------------

reader_landmask = reader_global_landmask.Reader(
... extent=[2, 8, 59, 63])
13:43:39 DEBUG: Adding new variable mappings
ERROR:root:could not verify read permissions for group and others on landmask.
Traceback (most recent call last):
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift_landmask_data-0.6-py3.8.egg\opendrift_landmask_data\mask.py", line 77, in check_permissions
if not os.stat(self.lockf).st_mode & 0o777 == 0o777:
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\mevo\AppData\Local\Temp\landmask\.mask.dat.lock'

[knutfrode]

Hi,

This seems to be a recent bug for Windows, related to file permissions for the landmask.
Can you try to delete any files in the folder C:\Users\mevo\AppData\Local\Temp\landmask\ (or simply the whole folder) and try again?

If it does not work, I will have a closer look at the code in a few days.

[mevo-creator]

Hi,
Thank you for prompt reply. i am getting the below error after removing the folder.

reader_landmask = reader_global_landmask.Reader(
... extent=[2, 8, 59, 63])
ERROR:root:fcntl not available on this platform, concurrent generations of landmask (different threads or processes) might cause failing landmask generation. Make sure only one instance of landmask is running on the system the first time.

[mevo-creator]

Also need to inform that after deleting the landmask folder when i run the command "reader_landmask = reader_global_landmask.Reader(extent=[2, 8, 59, 63])", the landmask folder is created again and i can also see the file mask.dat file in the folder.

[gauteh]

Hi, the landmask is probably generated as it should. It is a message notifying that on windows it is not possible to lock the landmask generation. Normally this is only shown once. I think the permissions message is shown every time (on windows), but it does not stop the reader.

[gauteh]

Fixed the landmask error message in OpenDrift/opendrift-landmask-data@a7479b0, but have not made a new release yet.

[mevo-creator]

Hi, thanks for the reply. Can i somehow use the Opendrift version where this error is fixed but not release yet?

[mevo-creator]

When trying to run some examples, I'm getting the below error it says "FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\2915803\AppData\Local\Temp\landmask\.mask.dat.lock'". Do you still think that landmark should be generated as usual? Please let me know about this so that i can ignore the error for future simulations. Thanks!

17:01:31 INFO: Adding a dynamical landmask with max. priority based on assumed maximum speed of 1.3 m/s. Adding a customised landmask may be faster...
ERROR:root:could not verify read permissions for group and others on landmask.
Traceback (most recent call last):
File "C:\Miniconda\envs\opendrift\lib\site-packages\opendrift_landmask_data-0.6-py3.8.egg\opendrift_landmask_data\mask.py", line 77, in check_permissions
if not os.stat(self.lockf).st_mode & 0o777 == 0o777:
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\2915803\AppData\Local\Temp\landmask\.mask.dat.lock'
17:01:36 INFO: Using existing reader for land_binary_mask
17:01:37 INFO: Moving 1 out of 2000 points from land to water
17:01:37 INFO: Oil-water surface tension is 0.022288 Nm

[gauteh]

Yes, you can ignore it. The error will disapear in the next version. The other error about the missing nc file you have to fix somehow. You can also use the unreleased version, but it is not necessary. See the docs for an example. You would have to do the same for the opendrift-landmask-data package. ons. 15. jul. 2020, 17:50 skrev mevo-creator <notifications@github.com>:

…

When trying to run some examples, I'm getting the below error it says "FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\2915803\AppData\Local\Temp\landmask\.mask.dat.lock'". Do you still think that landmark should be generated as usual? Please let me know about this so that i can ignore the error for future simulations. Thanks! 17:01:31 INFO: Adding a dynamical landmask with max. priority based on assumed maximum speed of 1.3 m/s. Adding a customised landmask may be faster... ERROR:root:could not verify read permissions for group and others on landmask. Traceback (most recent call last): File "C:\Miniconda\envs\opendrift\lib\site-packages\opendrift_landmask_data-0.6-py3.8.egg\opendrift_landmask_data\mask.py", line 77, in *check_permissions* if not os.stat(self.lockf).st_mode & 0o777 == 0o777: FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\Users\2915803\AppData\Local\Temp\landmask\.mask.dat.lock' 17:01:36 INFO: Using existing reader for land_binary_mask 17:01:37 INFO: Moving 1 out of 2000 points from land to water 17:01:37 INFO: Oil-water surface tension is 0.022288 Nm — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#337 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAN366FA45POI3JMH26KJ3R3XF45ANCNFSM4O2PBTNQ> .

[mevo-creator]

Thanks so much...appreciate your help!
I'm planning to use Opendrift for simulation of polymers (so basically plastic) in the Northsea, so i plan to use Plastdrift. My plan is to first go through the tutorials and examples in the gallery to understand how model works and then use it for my application. I would like to run simulation for 3-5 years after discharge of polymers, hope it is possible to do this in Opendrift(Plastdrift). It would be helpful if you guys can guide me a bit along the way!

[knutfrode]

Yes, this is all possible with PlastDrift module. But you will need a lot of input/forcing data for such a long term simulation.

I believe the other problem with the missing netCDF-file is that the file is not in your work folder. This specific file may be accessed with absolute path as in several of the example scripts, e.g. https://opendrift.github.io/gallery/example_generic.html#sphx-glr-gallery-example-generic-py

[mevo-creator]

Hi,
Thank you for your reply. I've some queries, it would be extremely helpful if you can guide me through them as per your convenience.

1. I understand the need for a large amount of forcing data. Our plan is to simulate the discharge of polymers from near Stavanger/Bergen and track them between 1 year to 5 years or possibly more in the North Sea and the Norwegian Sea. It seems like they will end up somewhere in the Barents sea but not exactly sure. I found some input/forcing data on met.no (https://thredds.met.no/thredds/catalog/fou-hi/norkyst800m-1h/catalog.html), however, I think I need to combine these files for a long time scale. is this the best approach, it seems like the file size is reasonably large and it might become too big of a file for 3 years simulation. Please suggest the best approach.

2. The polymers are discharged at a certain depth and therefore they will get distributed in the water column at very low concentration (perhaps in the range of nanogram/liter). In the configuration of the PlastDrift module, there are some configurations for vertical mixing. So is it possible to track polymers in 3 dimensions where when we seed the particles it gets randomly distributed in the water column? Also, I wonder what is the maximum number of particles that we can seed. Please let me know about this.

Finally, I must say that I was inspired by your presentation with some of your colleagues on "Simulated pathways of microplastics into the Arctic Ocean" that I found online. I saw some results for the distribution of plastics after 1-3 years. So I believe it could be possible to do this :)

Just would like to add that in case if it is not realistic to run the simulation for such a long time, it would also be helpful to run the simulation for a shorter time scale and in two dimensions.

Thanks again!
P:S I just deleted some of the text, as I felt it became too long.

[mevo-creator]

Hi, can someone from the team please share their thoughts regarding the possibility of using Opendrift for long time scale simulation in line with my above queries...thanks so much!

[knutfrode]

Hi,

Here is a template script in the direction of what you need:

from datetime import datetime, timedelta
from opendrift.models.plastdrift import PlastDrift

o = PlastDrift(loglevel=0)
o.list_configspec()  # to see available configuration options

# Analytical mixing is faster than 'randomwalk'
o.set_config('vertical_mixing:mixingmodel', 'analytical')

# If most particles are close to surface, we may save time by only reading
# upper layers (e.g. 3m) from ocean model, and extrapolating below
o.set_config('drift:truncate_ocean_model_below_m', 3)

# SVIM current 1960- 30 Sept 2019
# GFS winds 6 May 2011 - present
o.add_readers_from_list([
    #'https://thredds.met.no/thredds/dodsC/sea/norkyst800m/1h/aggregate_be'
    'https://pae-paha.pacioos.hawaii.edu/thredds/dodsC/ncep_global/NCEP_Global_Atmospheric_Model_best.ncd',
    'https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg'
    ])

# Seeding some particles
seed_time = [datetime(2012, 1, 1, 0), datetime(2012, 1, 2, 0)]  # Seeding over two days
end_time = datetime(2012, 1, 3, 0)  # Simulating for 3 days

o.seed_elements(lon=2, lat=60, number=3000, z=-20, time=seed_time,
                terminal_velocity=.01  # 1 cm/s as default
                )

o.run(end_time=end_time, outfile='plast.nc',
      time_step=timedelta(hours=.5), time_step_output=timedelta(hours=1))

o.animation_profile()
o.animation()

NorKyst covers the time from Sep 2016 until now, but not the Barents Sea. I believe the best sources for your purpose (as in example above) are the SVIM ocean model (1960-2019) for currents, and the NCEP GFS atmoshperic model (6 May 2011 until present) for winds.
Thus there are some years of overlap, and you can e.g. start your simulations 1 Jan 2012.
This is the same forcing data as was used for the Arctic simulation you mention.

The number of particles and duration is only limited by the memory available on your computer. OpenDrift flushes to disk every 100 time steps as default, and thus it is feasible to use ~million of particles over many years. The only challenge is that the output files are getting very large, and it may not be possible to re-import the output file into memory, in able to use the built-in plotting/analysis methods.
Unless you want to make your own analysis methods to handle huge datasets, a solution is to store the output data at larger intervals. E.g. for your purpose it should be sufficient to store output every 10 days, or even more seldomly. Note that the calculation time step should not be much larger than 1 hour, but this is independent.

You could play with the example script, and then increase the length and number of particles after confirming that output is as you like.

The terminal_velocity property of the particles would affect the drift, where higher velocities (i.e. larger particles) will drift closer to the surface (with Stokes and windage). The depth of seeding is probably not very important for such long term simulations, as particles will be mixed vertically very quickly after release.

[mevo-creator]

Hi @knutfrode , Thank you so much for detailed explanation of my query and providing me the example script to practise the simulations. I've tried running the script and it works well. I would like to learn more about in-built function and configurations for plastdrift model.
For instance like you mentioned that analytical is faster than random walk, similarly there is another configuration: vertical_mixing:diffusivitymodel [windspeed_Large1994] option('environment', 'stepfunction', 'windspeed_Sundby1983', 'windspeed_Large1994', 'gls_tke', default='windspeed_Large1994'). So, what does different alternative mean in the diffusivity model and how it may affect the output from simulation and so on for other configurations as well. Do we have some documentation for PlastDrift model where these functionalities are explained? Please let me know about this.

[knutfrode]

The vertical diffusivity determines the degree to which elements are mixed upwards and downwards due to turbulence (random walk scheme as described in Nordam2019), partly counteracting buoyancy which is normally upwards/positive for plastics (but you may also specify negative buoyancy as is normal for e.g. sediment drift).

The value of vertical diffusivity is ideally acquired from an ocean model, in which case you should specify
o.set_config('vertical_mixing:diffusivitymodel', 'environment')

However, this parameter is not always available in ocean model output files, and in this case it may be parameterized from wind (higher wind means more waves and thus vertical turbulence/mixing). E.g. it is included in the NorKyst model (with limited spatial and temporal coverage), but not in the SVIM model as in the example above.
The parameterisation from wind based on Large1994 is the default for PlastDrift, and should be a good alternative to using data from the ocean model. If using the NorKyst model instead, you may experiment with using environment instead.

sundby1983 is a simpler/older version of Large1994, where there is no depth dependence of the diffusivity. stepfunction is mainly for concept studies (idealised pycnocline), and is not recommended for your case. gls_tke is a special case when using output from ROMS ocean model, where diffusivity is stored as other parameters from which diffusivity may be derived.

This example script illustrates the vertical variation and wind speed dependence of diffusivity for the various parameterisations.

The buoyancy is given by the terminal_velocity that you may specify at time of seeding. This is a very important parameter for your simulations, and you should aim to find the correct values for your polymers. Instead of providing a scalar terminal_velocity for all elements as in example above, you may also provide different values for each element.

[mevo-creator]

Thank you for explaining different functions for vertical diffusivity and sharing the paper about this concept...I'll try out 'environment' using Norkyst model. I need to bring a couple of points into your notice about what we are trying to do and if it can be achieved by PlastDrift model.

1. I understand the importance of 'terminal velocity' as mentioned. However, I need to inform you that the polymers we are dealing with are Hydrolysed Polyacrylamide (HPAM) which are used for enhancing oil recovery. So, these HPAM based polymers are completely 'water-soluble' and not 'suspended'. They increase the viscosity of water when dissolved in the water. When this solution (polymer with water at higher viscosity) is injected in the reservoir it increases oil recovery by better sweeping the reservoir pores. there is a possibility that these polymers might also get a discharge in the marine environment. So if these polymers are discharged in the marine environment, they will not be 'suspended' but completely 'dissolved' in the water at very low concentrations (perhaps in the range of nanogram/litre, a bit similar to micro/nano plastics). Do you think PlastDrift would be suitable for such applications? Please let me know your opinion. I believe if PlastDrift can be used to track microplastics and nano plastics than it can also be used for polymers as well (both are almost similar) but would like to know your opinion.
2. If we can use PlastDrift (I really hope so), the plan is to see the extent of exposure (geographical area/volume of water) of these polymers in the Northsea and further up. Also to see if there is an accumulation of polymers in any particular area within the ocean (something like "Garbage Patch" that are found for plastics, there is one in the Arctic)...So would like to see when polymers are discharged from a different area in the NorthSea, do they all come together in some particular area and increase the concentration. I guess this can be achieved by running different simulations over time and see if all polymers end up somewhere in the same area.
3. From an environmental perspective, the positive thing about these polymers is that they are not 'persistent' like normal plastics, that means they will degrade but in the time scale of some years. (3-5 years even earlier if they are exposed to more sunlight)...so it would be interesting to see how and where these polymers will end up before they eventually degrade.

Please share your thoughts about this as per your convenience....thanks so much.

[mevo-creator]

Hi @knutfrode
Is above query something that you can guide me with. I understand that the results might carry some uncertainty for polymers that gets dissolved in the water, but we can account for this uncertainty by stating some assumption. However, on a broad level, if I can use PlastDrift model to simulate something like I’ve described above. Thanks for your help.

[knutfrode]

If the polymers are dissolved in water, you should instead use the OceanDrift model, which is the parent class of the PlastDrift model.

You should then perhaps also leverage on the restriction to only read the upper few meters of the ocean, as polymers may be found at any depth. However, this also means that the simulation will take quite a bit longer, as more ocean model data has to be read/ingested. On the other hand, you may in this case skip the windage and Stokes drift, which only applies to the upper meter or so. And also you may skip vertical mixing, which is here less relevant. So I might suggest something like this (not tested):

from datetime import datetime, timedelta
from opendrift.models.oceandrift import OceanDrift

o = OceanDrift(loglevel=0)
o.list_configspec()  # to see available configuration options

# Skip vertical turbulent mixing and Stokes drift
o.set_config('drift:vertical_mixing', False)
o.set_config('drift:stokes_drift', False)

# SVIM current 1960- 30 Sept 2019
o.add_readers_from_list([
    #'https://thredds.met.no/thredds/dodsC/sea/norkyst800m/1h/aggregate_be'
    'https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg'
    ])

# Seeding some particles
seed_time = [datetime(2012, 1, 1, 0), datetime(2012, 1, 2, 0)]  # Seeding over two days
end_time = datetime(2012, 1, 3, 0)  # Simulating for 3 days

o.seed_elements(lon=2, lat=60, number=3000, z=-20, time=seed_time)

o.run(end_time=end_time, outfile='polymers.nc',
      time_step=timedelta(hours=.5), time_step_output=timedelta(hours=1))

o.animation(color='z')
o.animation_profile()

With these conditions (ie. flow is particle-independent), I would believe the outcome is already given: the water/polymers from anywhere in the Norwegian seas will generally drift nortwards, where much of it will enter the Barents/Arctic Seas, where it may reside for some years, before eventually flowing southwards again in the Fram Strait. But there will be temporal variations in the drift, so you should probably release particles during your whole simulation period.

[mevo-creator]

Hi @knutfrode , Appreciate your quick and detailed response. I'll run the script and come back to you for further doubts..!!

[mevo-creator]

Hi @knutfrode , Thanks again for your help. I ran the script provided by you and it worked well. However all the particles gets seeded at 20 m depth (I think because of z = -20 here o.seed_elements(lon=2, lat=60, number=3000, z=-20, time=seed_time)). The polymers will be randomly distributed along the depth and therefore based on this example, I changed the code as below and this also worked well. 1000 particles got seeded and drifted for 30 days.

from datetime import datetime, timedelta #
import numpy as np
from opendrift.models.oceandrift import OceanDrift
from opendrift.readers import reader_netCDF_CF_generic
o = OceanDrift(loglevel=0)
o.list_configspec() # to see available configuration options
o.set_config('drift:vertical_mixing', False)
o.set_config('drift:stokes_drift', False)
o.add_readers_from_list([
#'https://thredds.met.no/thredds/dodsC/sea/norkyst800m/1h/aggregate_be'
'https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg'
])
z = -np.random.rand(1000)*50
seed_time = [datetime(2012, 1, 1, 0), datetime(2012, 1, 2, 0)]
end_time = datetime(2012, 1, 30, 0)
o.seed_elements(lon=4.8, lat=60.0, z=z, radius=0, number=1000,
time=seed_time)
o.run(end_time=end_time, outfile='polymers.nc',
time_step=timedelta(hours=.5), time_step_output=timedelta(hours=1))

However, when I tried to seed 10000 particles with the same code as above, there was an error as below. There might be many polymer particles as the concentration will be in nanogram/litre. therefore would like to seed more particle than 1000. Also, randomly distributed throughout the depth. Can you please guide me if the above code is the right approach for randomly seeding particles across the depth and for the higher time period as well. Also, if it is a right approach how can we resolve the error below to seed more particle.

02:53:09 DEBUG: Parsing variable: wro
WARNING:root:https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg is not a netCDF file recognised by OpenDrift: 'Reader' object has no attribute 'lon'
02:53:09 INFO: Opening dataset: https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg
02:53:09 INFO: Opening file with Dataset
02:53:09 WARNING: Vtransform not found, using 1
02:53:12 INFO: Read GLS parameters from file.
02:53:12 INFO: Making Splines for lon,lat to x,y conversion...
02:53:30 DEBUG: Setting buffer size 111 for reader roms native, assuming a maximum average speed of 5 m/s.
02:53:30 DEBUG: Adding new variable mappings
02:53:30 DEBUG: Adding method!
02:53:30 DEBUG: Reader initialised: roms native
02:53:30 DEBUG: Setting buffer size 24 for reader roms native, assuming a maximum average speed of 1 m/s.
02:53:30 DEBUG: Running lonlat2xy in parallel, using 2 of 4 CPUs
02:53:30 WARNING: Parallelprocessing failed:
02:53:30 WARNING: Can't pickle local object 'BaseReader.lonlat2xy..get_x'
02:53:30 WARNING: The simulation stopped before requested end time was reached.
02:53:30 INFO: ========================
02:53:30 INFO: End of simulation:
02:53:30 INFO: cannot unpack non-iterable NoneType object
02:53:30 INFO: Traceback (most recent call last):
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift\models\basemodel.py", line 2250, in run
self.get_environment(self.required_variables,
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift\models\basemodel.py", line 892, in get_environment
len(reader.covers_positions(
File "C:\Users\mevo.conda\envs\opendrift\lib\site-packages\opendrift\readers\basereader.py", line 904, in covers_positions
x, y = self.lonlat2xy(lon, lat)
TypeError: cannot unpack non-iterable NoneType object

02:53:30 INFO: 'The simulation stopped before requested end time was reached.'

Simulation aborted. 'The simulation stopped before requested end time was reached.'

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\mevo.conda\envs\opendrift\lib\multiprocessing\spawn.py", line 107, in spawn_main
new_handle = reduction.duplicate(pipe_handle,
File "C:\Users\mevo.conda\envs\opendrift\lib\multiprocessing\reduction.py", line 79, in duplicate
return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid

(opendrift) C:\Users\mevo>

[knutfrode]

There seem to be a Windows-issue(?) with the parallel processing. I will try to reproduce.

[knutfrode]

I could not reproduce this problem, so it seems to be a Windows issue.
Some parallel processing failed, but in this case OpenDrift should fall back to sequential processing. The fallback did however not work, but is now fixed.

This page might explain why the parallell processing failed on your machine:
https://stackoverflow.com/questions/52265120/python-multiprocessing-pool-map-attributeerror-cant-pickle-local-object

I implemented the suggestion there (3be2199), so if you now update your OpenDrift (git pull) it should hopefully work.

Since there is a new dependency in the meantime, you also need to update your conda environment. Updating directly with conda often fails, so I recommend deleting and creating anew with:

conda activate
conda env remove -n opendrift
conda env create -f environment.yml 
conda activate opendrift

[mevo-creator]

Hi @knutfrode , Thanks so much for the solution. I did a git pull and updated environment as you suggested and I was able to run the script for 10000 particles for 30 days. After git pull, the parallel processing fails but it falls back to single processor Multiprocessing has previously failed, reverting to using single processor for lonlat -> xy.

Now I wonder what could be the best alternative to simulate the same set of particles for a year or more than that. I remember your suggestion of storing output every 10 days (perhaps I can do it 30 days as well). But would it be possible to simulate same set of particles? for instance, now I have run this script for 30 days when I will run the script again for next 30 days, I need to seed the particles again and therefore previous particle trajectories will be lost somehow (even if it is stored in the output file). Please guide me how can we best approach longer time scale without disturbing particle trajectories from the previous simulation.

I guess the comment you made previously (please see below), will also play some role in reaching to solution for longer time scale.

But there will be temporal variations in the drift, so you should probably release particles during your whole simulation period.

[knutfrode]

You need to do a single, long simulation in able to track the same particles over a longer time.

You could e.g. seed particles over 2 years, and simulate for 3 years (i.e. the last year no more particles are released).
I guess storing output every 10 days should be ok. If you are not using more than 10.000 particles, you should then still be able to use OpenDrift to open, plot and analyse the results. If output file is too large for memory, you need to make own analysis/plotting methods.

It is not a big problem that the parallelisation does not work, as this is just a minor part of the simulation. The most time consuming part is to read the ocean model data, so a 3 year simulation probably takes several days.

I recommend storing the log to a file, with

o = OceanDrift(loglevel=0, logfile='simulation.log')

[mevo-creator]

Hi @knutfrode , Thank you for your answer. I was able to run and plot the results for 1000 particles and 30 days. It takes some time for making .gif and mp4 file for animation (around 20-25 minutes). I think it might take a long time for more than 30 days to plot the results.

1. Can you please guide me where exactly should I change the code to store output every 10 days as suggested by you, perhaps that can reduce the time? (I used this code for running the model now o.run(end_time=end_time, outfile='polymers.nc', time_step=timedelta(hours=.5), time_step_output=timedelta(hours=1)))
2. Also, how is it different to seed the particles over two years rather than seeding it for instance within a month? Is it because it may take more time for the model to read the trajectories of a higher number of particles for a long time (more data generated) compared to reading the trajectories of relatively lesser number of particles over a long time. Please let me know if my understanding is correct or there is some other reason for seeding particle over two years.
3. Also, when I run the script for 30 days, I'm getting the below error but the script continues to open dataset, finish simulation and plot the results, so is this error something that we need to worry about?
   WARNING:root:https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg is not a netCDF file recognised by OpenDrift: 'Reader' object has no attribute 'lon'
   19:50:16 INFO: Opening dataset: https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg

Please guide me about this as per your convenience. Thanks so much!

[knutfrode]

The time to conduct a simulation is approximately proportional to the total simulation duration. The size of the output file is proportional to number of particles and number of output time steps. The time to make an animation is proportional to the number of output time steps, and might thus be faster for a long simulation with 10 days output:
time_step_output=timedelta(days=10)

Seeding duration does not affect performance, only the number of particles.

The mentioned warning can be ignored.

[mevo-creator]

Hi, Appreciate your quick response...thank you.

[mevo-creator]

Hi @knutfrode , I ran simulations for 1 and 2 years for 1000 particles, the simulation runs ok. My primary goal behind running these simulations over a longer time scale is to get some idea regarding "accumulation" of the particles in a particluar area for a long time.

From one of your previous comment as quoted below,

With these conditions (ie. flow is particle-independent), I would believe the outcome is already given: the water/polymers from anywhere in the Norwegian seas will generally drift nortwards, where much of it will enter the Barents/Arctic Seas, where it may reside for some years, before eventually flowing southwards again in the Fram Strait. But there will be temporal variations in the drift, so you should probably release particles during your whole simulation period.

I can reconcile the drift of particles. For instance, from simulation over 2 years, majority of the particles end up in the Barents sea and some stay along the coastline. However, the particles are scattered all around, as we can see from the results below and perhaps bit difficult to say something concrete about accumulation.

The objective behind trying to see the "accumulation" of particles and/or "duration of accumulation" is to get some prediction about risk related to increase in concentration of polymers in a particular area when we have a continuous discharge. From the results below, we can say something about accumulation but perhaps not something precise.

So I need to ask two things

1. If there is some other configuration/approach that can help us to get a more precise idea about accumulation.
2. Also, what is the default size of the particles and is it possible to change the particle size somehow. I was just thinking if we can make particles smaller then may be it is bit more easier to get an idea about accumulation, but not sure about this.

I understand that the drift of particles will be quite random, but just wondering if there is some other function/configuration within the model that can help me to come close to what I'm trying to achieve in terms of accumulation.

Can you please comment on this as per your convenience, thank you.

[mevo-creator]

Hi, Similar to the above 5 years simulation, now I am trying to run the simulation for 10 years. I check the log file daily and got this error Error.txt today. The model was running ok until step 15013, however, some error occurred at step 15014. After this error, there is no interpolation in log also the number of missing elements disappeared suddenly from the previous step. Also, this error now continues to appear in every step with no interpolation showing up. Is it advisable to continue the simulation with this error?

[mevo-creator]

Hi @knutfrode , can you please look into the error?

[knutfrode]

The error should indicate that data is missing for the given times. The simulation can continue, but you will only get fallback values (0 wind, 0 currents).

You could try again to start a simulation at that specific time, to see if the same happens again. If so, there seem to be a hole in the SVIM dataset.

[mevo-creator]

ok. I understand. Thank you for the suggestion, I'll try to start the simulation at that time and see how it goes.

[mevo-creator]

Hi, Assuming there is a hole in the dataset, I changed the timescale and shifted it from 2000 to 1990 in another simulation. For this simulation, the same error started appearing from step 19190 (26/July/1996), please see the file
Error file.txt. Following your suggestion, a simulation for a shorter duration was started (for the time scale when error starts apperaing) from 25/7/1996. To my surprise this simulation was completed without any error of not able to find a netcdf file. Please see test.log

Can you please see what could be the error in this case, Thanks.

[mevo-creator]

Hi @knutfrode , can you please comment on what could be the issue in above case and if there is some solution for this, Thanks.

[knutfrode]

When there are holes in the dataset, the simulation will stop, unless you have added other backup readers, or provided fallback value for the given parameter. For OceanDrift, default fallback values of 0 are given for current, so that the simulation will not stop, but elements will not be moved until you have passed the hole.
If you start the simulation "after" the hole, it should be no problem, until you eventually reach the next hole.

[mevo-creator]

Yes I understand this. However as you had suggested earlier to start simulation at the time where the log start showing hole in the data set, just to perhaps confirm if there is really a hole in the data set. I did this cross-check by running another small simulation at the time when it starts showing hole in the longer simulation and surprisingly shorter simulation worked quite well. Does it not prove that there is perhaps no hole (unless I am missing out on something) Please see the log that I have shared above.

[knutfrode]

Yes, if you start the new simulation before/during the hole, and not afterwards, one would expect it to fail as well.
However, it may be that the "hole" in this case does not mean that data is actually missing, but that the thredds server was unavailable for some seconds. This may happen from time to time, and the risk is proportional to the length of the simulation.
However, if you are using a fallback value of 0, then it may for some applications be not critical if objects are not moving during a few hours of a simulation over several years.

[mevo-creator]

I see...I think that is the exact issue. The thredds become unavailable for long time scale simulations. I noticed that for the simulation that i started for 10 years and 15 years. So actually there is no hole but unavailability of the thredds server for some time. It seems like there is no solution to this issue. I can confirm that when I plotted the 10 years simulation the particles were frozen (because of fall back velocity 0) for the duration server was not available.

The purpose of running this long term simulation (10-15 years) is to see how long it takes for all water in the North/Norwegian sea to leave this area (volume flux, we can perhaps make some conclusion from the particles that are going out of coverage area (?)) and for freshwater to come from South. So that we can conclude that all the water in the North/Norwegian sea will be polymer-free after so and so years. do you have some suggestions to approach this?

[mevo-creator]

Hi @knutfrode , I am just sharing the result from the 10-year simulation through this link https://liveuis-my.sharepoint.com/:v:/g/personal/2915803_uis_no/EanqiCbZt4hIpCaFA-IoWjcB3T4vY2sC8podCfC1YcqqFA?e=pzzXQ4. Please see if it is possible to open it. The issue is that the model stops for about 3 years. It seems like a long time for the server to be unavailable.

[knutfrode]

I can see the movie.
The simulation uses very little time per time-step when there is no current data to read, so the server might have been unavailable for just a few minutes. But this can be seen from the time of the corresponding log-lines.

Anyway, these results can of course only be used until the data are unavailable. So if you need a longer simulation, you would need to re-run it. However, a continuous uptime of the thredds-server can unfortunately not be guaranteed.

[mevo-creator]

ok..I used o.run(end_time=end_time, outfile='10years.nc', time_step=timedelta(hours=3), time_step_output=timedelta(days=15)) to have smaller size of output file, but perhaps I can go little lower on the time step output. I'll re-run with shorter time step output and hope for the best. Thanks :)

[mevo-creator]

Hi @knutfrode , I am trying to run a new simulation with SVIM dataset, however below error is popping out when trying to read the dataset.
SVIM = reader_ROMS_native.Reader('https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "c:\users\mevo\opendrift\opendrift\readers\reader_ROMS_native.py", line 196, in __init__ self.times = [datetime.utcfromtimestamp((OT - File "c:\users\mevo\opendrift\opendrift\readers\reader_ROMS_native.py", line 196, in <listcomp> self.times = [datetime.utcfromtimestamp((OT - OSError: [Errno 22] Invalid argument

Can you please see what is the issue? Thanks

[knutfrode]

I cannot reproduce this problem, as the following works fine (though takes some time):

>>> from opendrift.reader.reader_ROMS_native import Reader
>>> r = Reader('https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg')

Can you try once more? If it does not work, you could try to update OpneDrift and dependencies with

$ git pull
$ conda env update -f environment.yml

[mevo-creator]

I did try again after updating Opendrift and dependencies (in two computers) and the exact error appears in both the system. I just edited the error because the model showed few more steps before showing the same error again.

>>> from opendrift.readers import reader_ROMS_native

>>> SVIM = reader_ROMS_native.Reader('https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg')
15:53:34 INFO: Opening dataset: https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg 15:53:34 INFO: Opening file with Dataset 15:53:34 WARNING: Vtransform not found, using 1 15:53:34 INFO: Read GLS parameters from file. Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Users\mevo\opendrift\opendrift\readers\reader_ROMS_native.py", line 196, in __init__ self.times = [datetime.utcfromtimestamp((OT - File "C:\Users\mevo\opendrift\opendrift\readers\reader_ROMS_native.py", line 196, in <listcomp> self.times = [datetime.utcfromtimestamp((OT - OSError: [Errno 22] Invalid argument

I was using the above command previously. But the command you suggested also doesn't work. Something is strange.

>>>from opendrift.reader.reader_ROMS_native import Reader
Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'opendrift.reader'

[mevo-creator]

Hi @knutfrode , I tried again today but the same error persists, could this be a windows issue? This command doesn't work either and gives the error as mentioned in the above comment.
>>> from opendrift.reader.reader_ROMS_native import Reader
>>> r = Reader('https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg')

was it possible to read the SVIM data set in windows in the latest Opendrift version (1.4.0), when you tried to reproduce the error? Could you please suggest the way forward? Thanks

I just removed and re-installed the model but still getting the same error when trying to read the dataset, please help as per your convenience :)

[mevo-creator]

Hi @knutfrode , The code stops running at line 196 in reader_ROMS_native.py which is related to datetime. It seems like this is the windows issue, I found this discussion on Python webpage This and this. It seems like something similar is going on in the error that I'm getting.

could you see some way to get around the error?

`

[knutfrode]

Yes, it seems to be a Windows issue.
It is a bit hard for me to debug this one since I have no Windows machine available, and also since days are quite busy.

But perhaps you can try in the meantime yourself to find a workaround (in reader_ROMS_native.py) according to discussions like the following?
usgs/libcomcat#184

[mevo-creator]

ok. I'll try to go through the discussion to find a way. I am a beginner in python so not sure how much I could achieve but I'll give it a try :) Thanks!

[mevo-creator]

Hi @knutfrode , The above issue related to windows got solved by itself in Opendrift version 1.4.2.

I've another query if you could guide me with. From literature studies, we have found that the sunlight (weathering) is responsible for breaking down of polymers over a period of time. However, the impact of solar irradiation changes with latitude and depth in the water column. We are trying to use Opendrift for analysing how the number of particles are changing with depth. Basically, trying to see at what depth the majority of particles end up after each time step. Is there any plotting/analysis tool within Opendrift that can give us output in terms of statistics (numbers) of particles at different depths? For instance, if 200000 particles are seeded and tracked for a year, after each time step what percentage of particles are between 0 and 10m, 10 and 20m and so on.

I did find an interesting plotting method o.animate_vertical_distribution(bins=30) in this example however this only works in Openoil and not in Oceandrift (the plotting takes a very long time in Oceandrift and not all time steps get plotted). I also tried some other plotting method that you'd suggested earlier for instance o.plot_property('z') or o.plot_vertical_distribution() however these methods generates plots of all particles that are seeded/tracked in the simulation and don't give statistics in terms of pure numbers. Is there some command/other approaches by which we can get numbers for eg percentage of total particles between 0 and 10, 10 and 20 and so on at different time steps? Please let me about this. Thanks

[knutfrode]

There are no specific methods to provide such statistics, but this should be fairly straightforward to do as post-processing from the history-array.

animate_vertical_distribution should work the exact same way for OceanDrift as for OpenOil, but it might be slow for many particles (as with any plotting/animation method).
It should also plot every output time step (which may be larger than calculation time step).

[mevo-creator]

ok. I would imagine so since the position of particles is stored in the output file at each time step. But how can we approach this post-processing from history-array. Do we need to import the position of particles from output file and make our own analysis method? it would be helpful if you could explain this briefly on how we could approach this?

I'll wait for more time for plotting animate_vertical_distribution

[knutfrode]

If you import the simulation to an object o, you can get the depths by
z = o.get_property('z')[0], with dimensions (output_timestep, particle_number)

This is also stored the same way as variable z in the netCDF-file.

From there you would need to make your own analysis method for what you want to achieve.

[mevo-creator]

Ok. I will try and implement this approach. Thank you for your guidance.

[mevo-creator]

Hi @knutfrode , I need to understand the resolution of the SVIM currents data set inorder to refer it in the publication ('https://thredds.met.no/thredds/dodsC/nansen-legacy-ocean/svim_daily_agg') that we are using as a reader in this simulation. I found this link SVIM that archives the data set. Also, this publication that explains how the SVIM data was generated.
Lienetal2013-EvaluationofaNordicSeas4kmnumericaloceanmodelhindcastarchiveSVIM1960-2011 (1).pdf

I need to know if this is the right link and publication corresponding to SVIM data set, please confirm this. Thanks

[knutfrode]

I can confirm that this is the right link and reference.
You may also mention that you accessed the data through thredds.met.no

[mevo-creator]

Thank you so much for quick response!

[gauteh]

Closing, re-open if needed.