Roadmap Meeting Schedule

[bemcdonnell]

The project is getting more popular which is incredibly exciting! I think we need to have a roadmap meeting soon to talk about recent history and future of the code base. Developers welcome.

Some topics to cover:

Recent History

1. Contribution workflows
2. Contributors License Agreements
3. Toolkit API
4. Contributions by USEPA for Regression testing within our CI frameworks.

Au Futur

1. SWIG support
2. Release workflow and scheduled releases
3. Thread safety
4. API expansion
5. Packaging and distribution
6. Improving the data model to Facilitate an input API.
7. Facilitating the needs of other sub-communities (like the 2D inundation models) through api
8. .... making everyone’s dreams come true in a “reasonable” way.

[goanpeca]

Hi @bemcdonnell maybe we could schedule a meeting to go over these issues and make it recurrent every... X days or something? and we can also arrange so that they get recorded.

Thoughts?

[samhatchett]

sounds good. how/when do you care to meet? will you doodle something?

I will also submit to the docket that I'm growing increasingly concerned about tooling overhead - the various CI integrations, enforcements, and conda business are all very confusing and seem to be causing issues for potential contributors. I don't have an easy answer to any of this, but that's why I bring it up.

[lrntct]

I have yet to contribute any code, but I'm interested to meet.
@samhatchett I believe that more documentation of the tools is needed, and that it should be easier for a new contributor to know where the project is going and why, and which kind of contributions are needed/welcome.

[abhiramm7]

Meeting seems like cool idea.

[bemcdonnell]

I will setup a Doodle for next week (I’m away from my desk until Monday). Until I get back; let’s shoot for late next week.

[michaeltryby]

Just to riff on @samhatchett concern a little bit. I think it is necessary for the project to incur some tooling overhead in order to achieve QA goals. Both projects are currently failing regression testing for various reasons. I agree that finding the right balance is key but I don't feel that we are there yet.

[goanpeca]

@samhatchett

I can write a document on the wiki to explain the ideas. I think the needs are, and pardon the use of we so loosely

0.) We want to have tooling to run regression tests and make sure on all platform we get the same results

By having all the containers and regression testing we can check for different platforms and different versions work and give the correct results

1.) We want SWMM to be cross platform

• The first step to get this going is done, and may need some tweaks, and the idea is to use CMake.
• Anyone should be free to build their own local version of SWMM with cmake and it should just work

2.) We want SWMM to be cross platform and have bindings to other language(s) so that scripting is easy

• At the moment the language of choice is Python.

• Bindings are built using CTypes from python and require "effort" every time the api changes

• The way Python is built distributed for scientific and academic (data science, machine learning, etc...) applications is with Conda, Miniconda, Anaconda and Conda-Forge. Conda also allows for the easy distribution and packaging of any type of application/binaries, (not only python stuff, that was the reason it exists). Tapping into this community is a big plus, because anyone using python for science and performance and ___________ already knows about conda (or should know about conda)

• Using SWIG opens the possibility of making SWMM available to other languages (if any developer feels like it) and opens the possibility of running testing from python via the bindings which is easier than trying to use GTest for running tests directly on C

3.) We want to make contributing code to SWMM easier

• We need more documentation to explain the possible workflows, but it boils down to,

  • A) Install all the stuff you need (VS... CMake, git clone etc..) and build and then make a PR.
  • B) Download conda and use conda-build which already takes care of dependencies because a conda-recipe is already available (CMake, VS, you can even install git with conda)

• Being able to use conda allows the user to have isolated system environments so they can work on different features that may require different compilers or dependencies.

4.) We want to have tooling to achieve QA in a more automatic way, linting code

It is a waste of a lot of peoples time to have to conform to a code style, in terms of use { here or there, separate this way, enforce line length, etc etc etc. We don't want to waste time reviewing this part of the code, nor we want to waste dev time on fixing this code manually so that is the idea behind CLang-Format tool. With conda is a simple command to install and use, manually we need to download CLang infrastructure and all that.

6.) We want to have the tooling and the tests to run unit testing for every PR that is pushed on every single platform

We need to have at least 2 CI systems, one for windows, where appveyor is our only choice and another system for OSX and CircleCI. CircleCI does not offer free OSX containers, so that leaves Travis as the only choice. Travis can handle Linux as well for free, but circle CI is much faster to run, so any problem is detected a couple of minutes after every commit. This would not be the case with Travis.

This requires writing tests. One way via SWIG is to add the python bidnings and use PyTest or UnitTest, which allow to write a whole test suite from python and not C, since SWIG will expose all the C code and not just a toolkit.

7.) We want to be able to distribute official releases in an easy an hopefully automatic way for all platforms and all compiler versions needed for binary compatibility

With conda-forge, the process of making a release is opening a new PR and change the tar file, hash and version number on the recipe, and then they are automatically built and released on anaconda.org/conda-forge.

We can also setup CI so that we can ourselves make the releases of exe etc, via CI, but this is not something we should do manually on someones home computer, that just does not cut it.

8.) We want to be able to distribute development, test, beta, RC candidates releases in an easy an hopefully automatic way for all platforms and all compiler versions needed for binary compatibility

Same here, the idea was to have a conda package built on every PR pushed to master or develop, that way anyone could use these libswmm-<some-version>-dev0 to run and test the latest installers.

Finally,

I will also submit to the docket that I'm growing increasingly concerned about tooling overhead - the various CI integrations, enforcements, and conda business are all very confusing and seem to be causing issues for potential contributors.

The problem is not conda or the tooling overhead, the problem is not having documentation around, These are the tools we use and This is why and This is How, this is partially my fault as I advocated for these tools, but I never got around doing a proper documentation. But the reasons were clear and discussed a long time ago, and at the moment @bemcdonnell and @michaeltryby were supportive of them.

"Proper" software development requires a lot of tooling, a lot of automation and a lot of documentation for both usage and development.

Just my dollar...

[lrntct]

@goanpeca thank you for the explanation.

The way Python is built distributed for scientific and academic (data science, machine learning, etc...) applications is with Conda, Miniconda, Anaconda and Conda-Forge. Conda also allows for the easy distribution and packaging of any type of application/binaries, (not only python stuff, that was the reason it exists). Tapping into this community is a big plus, because anyone using python for science and performance and ___________ already knows about conda (or should know about conda)

I might says that if you have a Python project, it must be on pypi. It is the official Python repository after all. From my little experience in a scientific context, my colleagues that use Windows use Anaconda, those that use Linux don't (I do not). I'm not against Conda, but it should not replace standard Python packaging imho.

B) Download conda and use conda-build which already takes care of dependencies because a conda-recipe is already available (CMake, VS, you can even install git with conda)

C) How to run the tests to make sure I don't break anything
D) How to create a new test for my new functionality

[michaeltryby]

@goanpeca Nice summary to which I would add:
0.) We want SWMM to build cleanly and give "correct" answers on all supported platforms.

[goanpeca]

@lrntct

I might says that if you have a Python project, it must be on pypi. It is the official Python repository after all.

We are talking about Swmm and not PySwmm. PySwmm is and will continue to be on pypi, that does not change. Conda has a source for recipes, so when building a python package we use pypi as source, and that source does not contain binaries or dlls, cause conda will have a separate package to provide that (libswmm and pyswmm depends on libswmm). For pypi only package the wheel will contain the binaries, but the recommended way will be to use conda, not pip.

From my little experience in a scientific context, my colleagues that use Windows use Anaconda, those that use Linux don't (I do not)

Depends on what they want to do and if they need system isolation or not, or/and if they care about being cross platform and being reproducible.

I might says that if you have a Python project, it must be on pypi.

Yes, and SWMM is not a Python project, so what is the point?

We can open an issue on https://github.com/OpenWaterAnalytics/PySwmm and discuss that there.

I'm not against Conda, but it should not replace standard Python packaging imho.

It does not replace it, it builds on top of it to handle all the cases that Pip and PyPi simply can't and opens up the space for packaging other stuff that is not python.

You can't do:

$ pip install libswmm
$ run-swmm --help

And doing:

$ pip install pyswmm
$ run-swmm --help

May or may not work depending on what the user has on their machine. It will not definitely run without issues if the bundled binary/library was built on a the compiler on @bemcdonnell 's computer (which is the current state of things) and if the user does not have the valid VS (version) runtime the user will need to manually install it

On the other hand, running:

$ conda install pyswmm  # This will pull libswmm plus other dependencies needed!
$ run-swmm --help

Will work on all machines, will download all the requirements and dependencies (VS runtime on windows, and libswmm package), and most likely will work out of the box, as all package were built using the right tools and right runtimes

C) How to run the tests to make sure I don't break anything

There no tests on SWMM besides the regression testing. Perhaps @michaeltryby could do a step by step instruction on how to run it?

tldr: Not yet possible or available

D) How to create a new test for my new functionality

SWMM does not have any testing, no unit tests, nothing. If we plan to migrate to C++ we could eventually use GTest (Google testing framework for C and C++). We could try to use GTest for current C code in swmm, but we will need to decouple the code in a better way. If we have the SWIG bindings ready we could try to write tests in python and do that.

tldr: Not yet documented in length?

---

@michaeltryby I would include that as part of the regression tests :-p, see point **5.) **

[michaeltryby]

@goanpeca you see it as point 5. I see it as point 0. It as the first most important thing that we should be doing right now.

[goanpeca]

There you go... updated ;-)

[bemcdonnell]

@goanpeca, swig will eliminate the need for any of us to build swmm locally. @lrntct has some nice workflows in mind and I think many of us have valuable perspectives.

Pyswmm could do its auto packaging on our CI servers for the respective platform and python version once code is merged into master. This will take away all dependency libraries like VCOMPxxx.dll. Instead of building with VS on my machine as we do today; a stable version of swmm can be specified to be wrapped up with a release version of a pyswmm wheel. If we let the pyswmm CI framework manage which release version to fetch from GitHub, we will have not synchronization issues since we can assert which version of the c code to build when it’s specified in a release on GitHub. On the other hand; if these projects are pushed into their distribution channels, independently; we can have synchronization issues for sure. I’d like to assert some version of swmm with some version of pyswmm.

Just want to put this project in context... most civil engineers know what computer programming is. Most programmers that I know use python instead of conda. For those who don’t understand programming very well; the simplest thing is for them to use python out of the box. Research groups and universities are different and so are developers. I foresee lots of people using this as a out of the box python package than a conda package.

This is a challenging topic for sure. Moreover, i am reading so many valuable perspectives! I appreciate everyone’s time and energy toward our common goals. Let’s step away and think about this again before our CI systems are changed (perhaps with the exception of appveyor). Let’s continue this discussion toward the end of next week in a group meeting.

[goanpeca]

@bemcdonnell I deleted your comments that talked about IPhone editing XD!

---

swig will eliminate the need for any of us to build swmm locally

Well.. not for develop if changes are introduced in the code. Unless you meant... ?

Conda packages are prebuilt binaries, so people do not build locally, they just install...

Most programmers that I know use python instead of conda. For those who don’t understand programming very well; the simplest thing is for them to use python out of the box.

This statement is simply not true for windows users, civil engineers or not.

The simplest thing is actually to install Anaconda which is a single click GUI installer instead of downloading python.exe and then numpy.exe and then scipy.exe and then networkx.exe from https://www.lfd.uci.edu/~gohlke/pythonlibs/ which is what people did for so long on windows until conda/anaconda/miniconda came along 4 years ago (yep its been that long already!)

[lrntct]

@goanpeca

Yes, and SWMM is not a Python project, so what is the point?

With the SWIG interface, libswmm could become a C python package, isn't it? Since swmm does not depends on anything beside libc, it might be reasonably possible to create a wheel package for libswmm. If it is too much of a hassle to have it work on many platforms, then I would understand not doing it.

There no tests on SWMM besides the regression testing. Perhaps @michaeltryby could do a step by step instruction on how to run it?

For the time being, a bit more information on how to run the regression tests (and maybe a small script to run them all) could be useful. At least to be sure to not break anything.

@lrntct has some nice workflows in mind

I have limited experience in multi-platform python packages with C extensions, and none with SWIG, so what I'm saying might be totally wrong.

[michaeltryby]

@goanpeca I use conda and think its great and I agree with many of your stated objectives. However, I am not convinced that Conda is the best approach for accomplishing them for swmmlib or epanetlib.

For example, regression testing has been integrated with Travis over on the EPANET side. This was accomplished using cmake, swig, python setup, and some shell scripting. Just 6 python packages are required and it only takes 1 min and 30 sec to boot, install, and run.

[bemcdonnell]

@goanpeca, with swig support we can issue a release here in the GitHub page.. then on the pyswmm CI servers, we can fetch the code from the release here, build it with swig, package it, and deploy to wherever. Our users would not need to generate any libraries locally. Our developers will generate libraries locally.

[bemcdonnell]

We will never have synchronization issues between the two projects If the pyswmm project is packaged with one single collection of files.

[michaeltryby]

@bemcdonnell Perhaps this will make more sense to other people once we swig the swmm toolkit.

[goanpeca]

@bemcdonnell

Our users would not need to generate any libraries locally.

This should not (ever) be the case. The development and release process is incorrect.

Our developers will generate libraries locally.

This is how it should always be. The development and release process is incorrect.

We will never have synchronization issues between the two projects If the pyswmm project is packaged with one single collection of files.

This is not related at all to the use of SWIG or the use of CONDA or the use of PIP. This is a problem of the current development ad release workflow.

---

@michaeltryby I understand, but what I suggest are 2 different things that get mixed up in the conversation.

1.) User Distribution Channels: I suggest it be the main and recommended distribution point for any SWMM user wanting to automate stuff with PySWMM. This means making the packages with conda. Specially if they are on windows they really should be using the conda ecosystem. We want to have more users and we want the software to work in tandem with all the other amazing tools from the scientific ecosystem. Saying pip is enough on windows is not true unless you are using python only libraries or using libraries that recurred to hacking the complete build tool chain so that it could work. We can even make a 1 click custom installer using conda and constructor for all OS systems and all the packages that make sense for SWMM users.

• Does this mean only conda? no, we can have multiple distribution channels.

2.) Development and dev channels: Since the main distribution channel I suggest is conda, the dev process could also use it and benefit from it as after it is in place. It would make it very easy for propective devs to just follow a couple of lines on the CLI and get the complete and the right development environment on any operating system, that having to install "just 6 dependencies". Bear in mind that for a lot of people python (not C or C++) is/was their first language so they will come here with that mindset after having used pyswmm and looking for ways to extend SWMM.

• Does it need to use conda?, not at all, but it could really make development easier for newcomers.
• Will it make it easier for you guys that are used to a different workflow? probably not and that is reflected on the "why do we need that". But for a newcomer there is no such complexity, but actually a much simpler workflow. It is not documented and it should, and that is on me.
• Maybe we ca use circle-ci to test all the conda stuff and run all the things that are not currently ran (linting for instance). Use appveyor and travis for the "normal workflow".

---

I will continue to argue in favor of 1, cause it is simply the best solution we have in the ecosystem that is, reproducible, cross platform, and friendly (specially on windows users!)

For 2, many projects do not use conda at all on CI, and the conda recipes resort to patching to fix things, and that is fine. I would however like to keep circle-ci running the conda flow as it already very fast and would allow to make sure things remain conda friendly.

I can maintain a set of instructions for Conda users and we can have another for non conda users, but I would like to see all the goals accomplished (not just the ones!) and not just some for the sake of speed on CI.

Also we need to really understand that for the SWMM world (and epanet world?) to work with the Python world, there are very specific things to follow:

https://wiki.python.org/moin/WindowsCompilers

and on https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/pull/107/files we are not doing that.

---

Again, I repeat the conda ecosystem is not a python thing only, there is R, Lua, Perl, many C/C++ system libraries that are already being packed this way.

https://anaconda.org/conda-forge
https://github.com/conda-forge/feedstocks/tree/master/feedstocks

3561 packages and growing...

[bemcdonnell]

@goanpeca, could you please provide us with a list of open source projects written in C where the conda workflows that you propose are being leveraged?

But from @bemcdonnell work on https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/pull/107/files

We are simply not doing that.

I have every intention of building with all of the necessary compilers. It is way more important that we pass the regression tests today.

Some of us have invested a lot of time and money into this project. We all need to be comfortable with the workflow. I'm not going to argue that the conda framework and workflow can't bring value; what I am suggesting is that some of us have a very nice workflow that we are happy with. It's portable and straightforward and reproducible with cmake. I like @samhatchett's philosophy with the epanet project. It's clean and organized thanks to the simplistic widely accepted approaches.

This statement is simply not true for windows users, civil engineers or not.

Do you know all the the civil engineers that I know? Without a doubt they're capable of anything! But we can't let their workflow get in the way of the work. I'm my experience; when I tell people that pyswmm is a python project, they google python and download it and follow my examples to get it working. More experienced coders will understand the value of a nice IDE like spyder and they will appreciate environments. We have entry level users and we have advanced users. I've stated an opinion here and I'm admitting it (please don't feel obligated to reply and quote me).

[samhatchett]

Our developers will generate libraries locally.

This is how it should always be... I don't understand what you are trying to clarify?

@goanpeca I think Bryant was referring to your comment "We can also setup CI so that we can ourselves make the releases of exe etc, via CI, but this is not something we should do manually on someones home computer, that just does not cut it." - So are you saying different things? I am confused.

Like Michael, I am convinced Conda is wonderful. Though I do not use it, even as a member of the scientific community. Maybe it's the only great package manager for windows (though i hear NuGet is great too), but we also have others for Mac (like Homebrew with over 4K packages, or MacPorts with over 20K) and for Unix there are countless package archives that work with dpkg. I'm not sure it's on my roadmap to be a package-manager evangelist.

While I understand being excited about what other people "Should" be using (and often I do try and evangelize for various causes), I usually only encounter resistance - especially when I use the "S" word.

I would advocate for being packager-agnostic. Really just simplify the scope of this repo so that the religious debates can take place elsewhere and we can focus on the C/C++ code and tests. Packaging and Distribution is SUPER IMPORTANT of course, but choosing a "primary" channel and conjoining SWMM development with those concerns feels like a distraction from the things we agree on.

Here is a hastily drawn figure to help illustrate the kind of project layout we've been migrating toward for EPANET.

[jrwalt4]

Hey guys, is this meeting open to the public, or by invite only? I'm based in Louisville, but would love to be a part of it.

I don't know most of you, though I met @samhatchett years ago when I was a grad-student at UK. I'm interested in the direction SWMM is headed and I'd love to contribute if possible; hopefully offer some perspective on how we use SWMM for our modeling/design projects at Stantec.

Thanks!

[bemcdonnell]

@jrwalt4, sure - we can open it up to some others! I’ll put a doodle up on this issue when I get back.

[goanpeca]

@samhatchett yes, that makes sense. I am in charge of conda packaging on that chart and I will take that part ;-). I can also offer additional instructions for those who might be already be using conda and conda-build.

Now, all this discussion I think stems from not having clear build instructions and test instructions for contributors.

We need to update the README.md to include that, which needs to include all dependencies needed with some base systems.

---

Suggested content @samhatchett, @michaeltryby @bemcdonnell. We need something like on the README.md, or as a CONTRIBUTING.md or BUILDING.md file.

I.a. Development environment (Normal Workflow)

A.) Windows

Install Dependencies

• Visual Studio C++ 10. Link to download
• Install CMake >= <VERSION> Link to download
• Python 3.4. Link to download
• Install the nrtest dependencies..... Links to download

Building

To build SWMM then run:

$ cmake -G "Visual Studio <VERSION>" -DCMAKE_INSTALL_PREFIX:PATH=output -DCMAKE_BUILD_TYPE:STRING=Release

Running

$ run-swmm --help

B.) Linux

Install Dependencies

• Build dependencies... gcc >= libgcc >=
• cmake
• Install the nrtest dependencies..... Links to download

On Debian (.deb) based systems you can use

$ sudo apt install ...

On Centos (.rpm) based systems you can use

$ sudo yum install ...

Building

$ cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE:STRING=Release ..
$ make -j $CPU_COUNT
$ make install

Running

$ run-swmm --help

C.) OSX

InstallDependencies

• Clang version >= ??? or
• gcc version >= ???
• cmake
• Install python and the nrtest dependencies..... Links to download or commands

If using Homebrew

$ brew install ...

Building

$ cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE:STRING=Release ..
$ make -j $CPU_COUNT
$ make install

Running

$ run-swmm --help

I.b. Development environment (If using conda)

A.) Windows

Install Dependencies

• Visual Studio C++ 8 for python 2.7. Link to download
• Visual Studio C++ 10 for python 3.4. Link to download
• Visual Studio C++ 14 for Python 3.5 and 3.6. Link to download

Then you can create a development environment with all the dependencies, activate and build from there.

$ conda create -n swmmdev cmake conda-build python=<2.7, 3.4, 3.5, 3.6> nrtest ... <any other thing needed for testing>

Activate the environment:

$ activate swmmdev

Building (with conda-build)

(swmmdev) $ conda-build conda.recipe
(swmmdev) $ conda install libswmm --use-local

Running

(swmmdev) $ run-swmm --help

B.) OSX and Linux

Install Dependencies

• Gcc... (provided as conda packages...)
• cmake (conda package)
• python (conda package ...)

Create environment:

$ conda create -n swmmdev cmake conda-build gcc libgcc python=<ANY VERSION> nrtest ... <any other thing needed for testing>

Activate the environment:

$ source activate swmmdev

Building (with conda-build)

(swmmdev) $ conda-build conda.recipe
(swmmdev) $ conda install libswmm --use-local

Running

(swmmdev) $ run-swmm --help

II. If you are developing for PySwmm

If you are developing for PySwmm you need to install these versions to work with how python is compiled on different platforms:

A.) Windows

Python 2.7

• Visual Studio C++ 9 for Python 2.7. Link to download
• Python 2.7. Link to download

Python 3.4

• Visual Studio C++ 10 for Python 3.4. Link to download
• Python 3.4. Link to download.

Python 3.5 and 3.6

• Visual Studio C++ 14 for Python 3.5 and 3.6. Link to download
• Python 3.5. Link to download
• Python 3.6. Link to download

B.) Linux

....

C.) OSX

....

III. Regression Testing

To execute regression tests run:

$ python <script...>

IV. Unit tests

To execute unit tests run:

$ python <script...>

[dbhart]

* > > Our developers will generate libraries locally. * > This is how it should always be... I don't understand what you are trying to clarify? * @goanpeca<https://github.com/goanpeca> I think Bryant was referring to your comment "We can also setup CI so that we can ourselves make the releases of exe etc, via CI, but this is not something we should do manually on someones home computer, that just does not cut it." - So are you saying different things? I am confused. As a developer for Python and EPANET, and as someone who deals with cyber security issues, I would suggest that: * For developers, who can build themselves, not having a CI that releases binaries is no big deal * For users who may want to use PySWMM or derived tools, having a binary that was built outside a CI environment would be a significant security concern. While it is true that a CI system might become infected, it is far more likely that it would be detected early, vs. an infection that came from a single computer. I mention this specifically because philosophically I would agree with previous statements that Users should never need to download a compiler. I’m a developer, I’m going to either build my own DLL’s / pyd’s, but I support customers that rely on us to provide binary installers – having the distribution be part of an automated process significantly decreases risk exposure. And I admit, my customer base is perhaps different from many here since I am focused on EPANET, but since I got all the e-mails (not sure how I got on this particular thread, but oh well (-: ) I thought I’d share that security focused perspective.

…

-- David Hart | e: dbhart+NOSPAM@sandia.gov<mailto:dbhart@sandia.gov> Sandia National Laboratories | Geoscience Research & Applications Group | Geochemistry Dept.

[goanpeca]

My apologies for being so pushy on this (conda) subject. I understand at the moment is not a good choice for CI or development workflow if someone has not used it before. I will be in charge of making the conda packages once we can sort out the nrtest regression tests on CI based on the work by @michaeltryby. Cheers.

[michaeltryby]

@goanpeca check out the feature-nrtest branch. nrtest is already configured to work under Travis. Just merge the branch.