Any plans for a Linux enabled version? #50

bhlevca · 2020-06-15T21:04:11Z

The current version in __init__.py is loading dlls. If the equivalent functionality can be replaced with shared libraries perhaps a Linux or MacOS version could be achieved.

It is possible that the Linux version could be made available sooner as there are already binaries for the cloud enabled Linux version of Mike Zero.

The text was updated successfully, but these errors were encountered:

bhlevca · 2020-07-02T19:34:48Z

If I am given some guidance I can try to implement this.

halvgaard · 2020-07-03T07:01:52Z

@bhlevca Great :)
We will hit summer break soon so might not be able to help much in July.
If you manage to setup a quick draft, you can PR your branch into master as a draft that we can follow and discuss and link to this issue.

bhlevca · 2020-07-03T12:23:02Z

@Rhadhi Thanks for your answer.
What kind of draft you have in mind?
I will need some information about the Linux shared libraries that contain the information needed by mikeio to handle the DFS files.
Also, I see there is a dependency on the .Net API, which I suppose is used in the Mike DLLs. Linux has some issues with that, but Mono development tools may help with that.

halvgaard · 2020-07-03T18:29:31Z

@bhlevca I think that @JesperGr mentioned that it is possible without Mono.

bhlevca · 2020-07-03T18:35:03Z

@Rhadhi, @JesperGr, that would be great. If I get some starting tips, I could try working on it.

bhlevca · 2020-09-29T15:49:49Z

@Rhadhi I am ready to try my hand at it. You told me above that it would be possible to do it without Mono. If I get a push in the right direction I will try to develop the glue code for this.

halvgaard · 2020-09-30T07:07:45Z

@JesperGr Any suggestions?

JesperGr · 2020-09-30T07:46:36Z

Right now the library is based on the MIKE Core .NET libraries, and utilizing the ease of the .NET API. That requires .NET integration in python.

I have no idea if the .NET integration in python works on Linux, or whether it can be made to work. I guess that would be the first thing to test - testing need not be based on MIKE Core assemblies, but just getting any .NET Standard library working in python could show if that is an option.

The alternative is to use the C API to DFS files. Using a C API is natively support by Python on both Windows and Linux. However, that will be a major rewrite of the code, since the C API it is quite different from the .NET API.

And then comes the issue with DHI Linux libraries, which are currently not publicly available, and are only build for a few distributions (e.g. known not to work on Ubuntu), so I am not sure that part is fully ready either.

ecomodeller · 2020-09-30T08:03:11Z

Even though it could be a significant effort, the best solution to be able to provide a working library on both Windows and Linux seems to use the C API.

Is there documentation and examples available for the C API?

JesperGr · 2020-09-30T12:04:17Z

The C API is not as well documented as the .NET variant. There is one external example at:

https://github.com/DHI/MIKECore-Examples/blob/master/Examples/C/DHI.MikeCore.CExamples/DHI.MikeCore.CExamples.cpp

That should cover most of what is required for dfs0+1+2+3. For dfsu, interpretation of the static mesh data will be required. Documentation for how DFS is used for dfsu files can be found here:

https://manuals.mikepoweredbydhi.help/2020/General/FM_FileSpecification.pdf

We do plan on updating the examples to also include at least dfsu - just not there yet.

The header file for all the C-methods are available in the latest version of the NuGet package DHI.DFS. There is some DHI-internal documentation of the C API, which I can also send on, if required. Otherwise we can find several examples internally within the DHI codebase.

The more tricky part is writing dfs files, where the C api methods needs to be called in a specific order.

Support for Res1D files will be difficult, because that one is highly dependent on .NET code on top of the DFS functionality.

bhlevca · 2020-10-01T04:08:51Z

I found out that pythonnet can be installed in Ubuntu

https://stackoverflow.com/questions/55058757/install-pythonnet-on-ubuntu-18-04-python-3-6-7-64-bit-mono-5-16-fails

using Pythonnet 2.5.1:

sudo apt update
sudo apt-get install clang
sudo apt install nuget
git config --global http.proxy http://my.proxy.address:port
/usr/bin/python3 -m pip install -U pycparser --user
/usr/bin/python3 -m pip install -U pythonnet --user

and this documentation about .Net on Linux

https://medium.com/wolox/how-to-create-your-first-net-e2223dedb74f

I am going to explore this path first, although I agree with @ecomodeller that the C-API solution more elegant and universal.

I will let you know what I find with my tests. I am working on OpenSuse not Ubuntu. and I had no issues installing pythonnet on it using pip/wheels.

ecomodeller · 2020-10-05T18:48:09Z

A list of things that would be possible with a linux version of mikeio:

Pre and postprocessing of data for models running on HPC clusters 💻
Docker (very general purpose) 🚢
Web applications 🕸️
Azure functions 🔢
Azure machine learning 🧠
Raspberry PI ❓

bhlevca · 2020-10-05T19:45:40Z

@ecomodeller Great list.

I hope to start working on this this week. Starting with testing Pythonnet on linux. I installed it but have not got a chance to test it

bhlevca · 2020-10-19T01:57:04Z

I had partial success with pythonnet on Linux

I started with something simple, to create a DFS2 file from some data.
I debugged a little and clr in mikeio __init__.py seems to load the reference just fine.
The code throws an exception in write()-> dfs.py ->_setup_header()->self._builder.CreateFile(filename)
The code and stack trace are below. It looks that an invoke method may not be possible to run in Linux.

I'l kike to know your opinions and if you have any suggestions.

[...]
dfs = Dfs2()
dfs.write(filename=dfsfilename,
           data=d,
           datetimes=Time_Frame,
           items=items,
           coordinate=coordinate,
           dx=dx,
           dy=dy,
           title="Wind 2018"
)

System.DllNotFoundException: eum assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) DHI.Generic.MikeZero.EUMWrapper.eumSetupLoadLinux(string)
  at DHI.Generic.MikeZero.EUMWrapper..cctor () [0x0002d] in <89a8313aa64241c293af856b1063ee2c>:0 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/software/SAGEwork/Calibration-LakeOntario/CFSR/Wind DATA/Wind_grib_to_dfs2.py", line 102, in <module>
    dfs.write(filename=dfsfilename,
  File "/software/SAGEwork/mikeio/mikeio/dfs2.py", line 296, in write
    dfs = self._setup_header(filename)
  File "/software/SAGEwork/mikeio/mikeio/dfs.py", line 152, in _setup_header
    self._builder.CreateFile(filename)
System.TypeInitializationException: The type initializer for 'DHI.Generic.MikeZero.EUMWrapper' threw an exception.
  at DHI.Generic.MikeZero.eumQuantity..ctor (DHI.Generic.MikeZero.eumItem item, DHI.Generic.MikeZero.eumUnit unit) [0x00014] in <89a8313aa64241c293af856b1063ee2c>:0 
  at DHI.Generic.MikeZero.DFS.DfsSimpleItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo itemInfo) [0x00035] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo itemInfo) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsDynamicItemInfo..ctor (DHI.Generic.MikeZero.DFS.IDfsDynamicItemInfo dynamicItemInfo) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.DfsDynamicItemBuilder.GetDynamicItemInfo () [0x00023] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.dfs123.Dfs123Builder.CreateBuilder () [0x0019f] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at DHI.Generic.MikeZero.DFS.dfs123.Dfs123Builder.CreateFile (System.String filename) [0x00000] in <95903b2b47bb416faac8f2fb84766e7b>:0 
  at (wrapper managed-to-native) System.Reflection.RuntimeMethodInfo.InternalInvoke(System.Reflection.RuntimeMethodInfo,object,object[],System.Exception&)
  at System.Reflection.RuntimeMethodInfo.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0006a] in <5514b7b0ec654df1b74df5da6e884f74>:0

JesperGr · 2020-10-20T06:48:05Z

Dear bhlevca

I think that is as far as you can get with what you have - which is actually good news.

The mikeio package depends on the .NET MIKE Core package, which again depends on native dll's. But the native dll's that you have (eum.dll, ufs.dll), they are all Windows, and will not work on Linux. So you get a
System.DllNotFoundException: eum

The good news, what you have shown is that the mikeio package can use the .NET MIKE Core package also on Linux. That was a question posted earlier in this thread.

Then we are back to the fact that the DHI Linux libraries are currently not publicly available, and are only build for a few distributions (e.g. known not to work on Ubuntu).

If you can describe exactly the steps you did in order to get this far, then maybe someone at DHI with access to the DHI Linux Libraries can pick this up to check if we can get all the way.

bhlevca · 2020-10-20T18:10:50Z

Dear Jesper,

I tried many things including with Mono and pythonnet installed with pip. However, my recommendations are the following:

.Net Core

Install .Net Core SDK ( preferably 5.0) for Linux with the install script (optional with the snap package, but that a bit quirky) from here dotnet script install
run ./dotnet-install.sh -c 5.0 (3.1 could also be used but I recommend 5.0)
Put the dotnet location in the user or system $PATH preferably in a profile.d script that's started at boot.

Pythonnet

Install pythonnet from github run git clone https://github.com/pythonnet/pythonnet
run python setup.py bdist_wheel --xplat (do not remove --xplat as it will use Mono on OSX and Linux instead)
run pip install --no-index --find-links=./dist/ pythonnet (it wil install dependencies as well)

mikeio

Write a python script that uses mikeio (I put mikeio in the python path as is changes often and I don't want to build the packages and install it every time)

I work on OpenSuse and my commands differ a bit, for example, I use python3 and pip3 instead of python and pip.

Please let me know if there is anything else you need to know.
I have access to the Linux shared libraries for Mike, but they have different names than the Windows DLLs and I would not know which ones to load and how to access them.

Bogdan

bhlevca · 2020-11-06T14:18:41Z

If I get some help finding the right Linux libraries where I can find the Windows equivalent classes I could try to write an __init__.py for Linux

ecomodeller · 2021-03-02T16:46:00Z

Just to let you know, that we will soon start the process of migrating away from using .NET and use a new Python version of the MIKE SDK.

This will enable us to use most of MIKEIO on Linux in a not too distant future.

benjaminh · 2021-03-15T12:50:05Z

Just to let you know, that we will soon start the process of migrating away from using .NET and use a new Python version of the MIKE SDK.

This will enable us to use most of MIKEIO on Linux in a not too distant future.

Great news !
Any ideas on when this future will occur (few weeks/months/years) ?

ecomodeller · 2021-03-15T12:53:27Z

The future is expected to arrive in a few months.

bhlevca · 2021-03-16T00:24:17Z

This is great news. This is going to be a long few months.
Is the new Python-based/version of MikeSDK available for our viewing? If yes, could you share a link.
Thanks

ecomodeller · 2021-05-04T15:09:24Z

We are getting closer...

bhlevca · 2021-05-04T15:58:27Z

Great news.

I checked out mikecore but I am not getting the Linux binaries. I build the package and installed it but it is still creating a mikeio directory in site-packages and not a mikecore as in your example.

Is there any way I can get access and start playing with it? I will make no complaints if it does not work :-)
Thanks

ecomodeller · 2021-05-04T17:47:26Z

The Python-based/version of MikeSDK (mikecore) will be available as soon as we have sorted out some technical and maybe also some licensing details.

The implication of this refactoring is that we will have to split the functionality of mikeio into two repos, one which doesn't need .Net (mainly dfs) and one that relies on .Net (Mike 1D).

ecomodeller · 2021-06-01T11:47:22Z

The Python version of Mike core (mikecore) is now available on Github.

In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box.

$ pip install https://github.com/DHI/mikeio/archive/mikecore.zip

jsmariegaard · 2021-06-01T12:16:00Z

In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box.
$ pip install https://github.com/DHI/mikeio/archive/mikecore.zip

Just a small note: This first .NET-free version of mikeio will read/write dfs0 slower than the old version. This will be fixed soon!

bhlevca · 2021-06-01T21:02:55Z

Thank you. Excellent work. I installed with "python setup.py <https://signaltk.online/click?redirect=http%3A%2F%2Fsetup.py&dID=1622581344395&linkName=setup.py> install" even though it is not mentioned in the Readme.md <https://signaltk.online/click?redirect=http%3A%2F%2FReadme.md&dID=1622581344395&linkName=Readme.md> file.

…

On Tue, 1 Jun 2021 at 08:16, Jesper Sandvig Mariegaard < ***@***.***> wrote: In order to give this new version of mikeio, you can install mikeio from this branch and it should hopefully work out of the box. $ pip install https://github.com/DHI/mikeio/archive/mikecore.zip Just a small note: This first .NET-free version of mikeio will read/write dfs0 slower than the old version. This will be fixed soon! — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#50 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABMRXWCPLTHMC24KLP7SCTTQTFQ5ANCNFSM4N6XSV3A> .

benjaminh · 2021-06-17T12:16:42Z

The URL provided above (https://github.com/DHI/mikeio/archive/mikecore.zip) seems to be broken, it returns a 404 error. Did it move elsewhere ?

jsmariegaard · 2021-06-17T12:19:07Z

@benjaminh , it's been merged in to the main branch: https://github.com/DHI/mikeio/archive/main.zip

ecomodeller · 2021-08-25T13:09:05Z

The Linux version works as expected!
The only thing that is different at the moment is read/write performance on large dfs0 files.

bhlevca · 2021-08-25T15:56:35Z

Many thanks for making this happen!

ecomodeller added the enhancement New feature or request label Aug 21, 2020

halvgaard added alternative deployment help wanted Extra attention is needed labels Sep 16, 2020

halvgaard assigned bhlevca Oct 1, 2020

ecomodeller closed this as completed Aug 25, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Any plans for a Linux enabled version? #50

Any plans for a Linux enabled version? #50

bhlevca commented Jun 15, 2020

bhlevca commented Jul 2, 2020

halvgaard commented Jul 3, 2020

bhlevca commented Jul 3, 2020 •

edited

halvgaard commented Jul 3, 2020

bhlevca commented Jul 3, 2020 •

edited

bhlevca commented Sep 29, 2020

halvgaard commented Sep 30, 2020

JesperGr commented Sep 30, 2020

ecomodeller commented Sep 30, 2020

JesperGr commented Sep 30, 2020

bhlevca commented Oct 1, 2020 •

edited

ecomodeller commented Oct 5, 2020

bhlevca commented Oct 5, 2020

bhlevca commented Oct 19, 2020 •

edited

JesperGr commented Oct 20, 2020

bhlevca commented Oct 20, 2020 •

edited

bhlevca commented Nov 6, 2020 •

edited

ecomodeller commented Mar 2, 2021

benjaminh commented Mar 15, 2021

ecomodeller commented Mar 15, 2021

bhlevca commented Mar 16, 2021

ecomodeller commented May 4, 2021

bhlevca commented May 4, 2021

ecomodeller commented May 4, 2021

ecomodeller commented Jun 1, 2021 •

edited

jsmariegaard commented Jun 1, 2021

bhlevca commented Jun 1, 2021 via email

benjaminh commented Jun 17, 2021

jsmariegaard commented Jun 17, 2021

ecomodeller commented Aug 25, 2021

bhlevca commented Aug 25, 2021

Any plans for a Linux enabled version? #50

Any plans for a Linux enabled version? #50

Comments

bhlevca commented Jun 15, 2020

bhlevca commented Jul 2, 2020

halvgaard commented Jul 3, 2020

bhlevca commented Jul 3, 2020 • edited

halvgaard commented Jul 3, 2020

bhlevca commented Jul 3, 2020 • edited

bhlevca commented Sep 29, 2020

halvgaard commented Sep 30, 2020

JesperGr commented Sep 30, 2020

ecomodeller commented Sep 30, 2020

JesperGr commented Sep 30, 2020

bhlevca commented Oct 1, 2020 • edited

ecomodeller commented Oct 5, 2020

bhlevca commented Oct 5, 2020

bhlevca commented Oct 19, 2020 • edited

JesperGr commented Oct 20, 2020

bhlevca commented Oct 20, 2020 • edited

bhlevca commented Nov 6, 2020 • edited

ecomodeller commented Mar 2, 2021

benjaminh commented Mar 15, 2021

ecomodeller commented Mar 15, 2021

bhlevca commented Mar 16, 2021

ecomodeller commented May 4, 2021

bhlevca commented May 4, 2021

ecomodeller commented May 4, 2021

ecomodeller commented Jun 1, 2021 • edited

jsmariegaard commented Jun 1, 2021

bhlevca commented Jun 1, 2021 via email

benjaminh commented Jun 17, 2021

jsmariegaard commented Jun 17, 2021

ecomodeller commented Aug 25, 2021

bhlevca commented Aug 25, 2021

bhlevca commented Jul 3, 2020 •

edited

bhlevca commented Jul 3, 2020 •

edited

bhlevca commented Oct 1, 2020 •

edited

bhlevca commented Oct 19, 2020 •

edited

bhlevca commented Oct 20, 2020 •

edited

bhlevca commented Nov 6, 2020 •

edited

ecomodeller commented Jun 1, 2021 •

edited