Software Engineering Best Practices in Science, the compiling optional tutorial

[asx-]

Title

Software Engineering Best Practices in Science, the compiling optional tutorial

Summary

While there was a plan to have an optional tutorial on "Software Engineering Best Practices 101 for Scientists", life happened and it was not put together in time for the event week. Instead, let's put it together now and give a short presentation at the end of the week.

Personnel

Joseph Gum, potential lead and secretary (note - this project could have a different lead or not happen if other projects I'm proposing are more popular)

Specific tasks

• Use a code linter and formatter
• Writing good comments
• Filling in github issues well (or JIRA, or Asana, etc if we must)
• Submitting good bug reports
• Reviewing code from others
• Linking extensions between different services, like Github and Slack, etc.
• Lots of things I'm probably missing...

Data sets and infrastructure support

List any external data sets, particularly large ones, that you need for the project, as well as other infrastructure support such as libraries that needs to be included in the Hub -- if the project will be worked on using Hub resources (vs local machine). Feel free to ping @help-infrastructure if you want to discuss these with the infrastructure team.

The problem

Do you have impostor syndrome with sharing your code? Do you know your code could be much better? Do you get cold feet filing bug reports online because you think you're doing something wrong? (You might actually have a very rare bug!)

Let's work through these issues together and become confident in doing so!

[Anujjain2579]

Would like to contribute more points -
We can include :
Defining functions for repetitive procedures.
Following standard (or more sensible) color combination in our graphs / plots.
Creating and Maintaining workflows for large collaborative projects.

-Anuj Jain(Student)

[abkfenris]

There was a least one video from SciPy that covered parts of this.

[asx-]

https://www.youtube.com/watch?v=oGzU688xCUs

The video he's talking about.

[asx-]

Some comments from friends of mine that are programmers at Big Tech:

1. Make it easy to do the right thing, and make it hard to do the wrong thing.
2. Write code for humans, let the language/framework/linters focus on optimization and style.
3. Too many layers of abstraction is terrible.
4. Make it reasonably easy to be tested.
5. Hide your implementations, provide public interfaces and abstractions. Register implemnetations via modules
6. Build for 98% of scenarios, not for the rare 2%
7. Make the hard easy to achieve, and the impossible hard to achieve.
8. Try to facilitate debugging through logs and metrics rather than having a full repro(dutcion) setup.
9. Don't waste time re-implementing the wheel.
10. If something doesn't need to be public, it's always internal.
11. Use a real SCM (Source Code Management) and CI/CD (Continuous Integration/Continuous Deployment)
12. Follow semver (semantic versioning)
13. If you publish bits as OSS, take a hard stance on which OSes, runtimes, etc you officially support. Otherwise you'll go crazy. And make your official support prominent on your README.

Also:

A codebase is like an investment. You can keep putting hours in and it'll grow, but it'll grow faster and faster the more you can make future things easier/faster to do, automate things, and make it easier for others to contribute.

Some of this is very specific and some of it we won't have time to cover, but some thoughts from people in industry that write code.

[asx-]

Another thing - developing the skills needed to work seamlessly in remote contexts, how to go about doing that, etc.