Move to a more appropriate language

[Minoru]

This is a thought that's been popping up in my mind time and again. I seem to forget half my arguments by the time I think of this again, so let me write this down and let's discuss it.

This is not a plan, and I definitely don't want to rewrite anything until I understood the limitations behind a few of the big problems that Newsboat faces: long startup, memory consumption, and how keys work.

Why rewrite

For Newsboat, I think C++ is a death by a thousand cuts:

• too low-level for the task at hand. Manual memory management (reference counting helps, but still is suboptimal), unsafe things like pointer aliasing, weak typing, inexpressive type system—this doesn't help us write robust software at all.
• lacks convenient build system. GNU Make doesn't understand dependencies between C++ compilation units; Makefiles are all custom, and common things (like builds with profiling) have to be implemented all over again. I tried CMake (see #14) and waf (didn't seem like much of an improvement over Makefiles).
• lacks convenient package management. (But this can also be an advantage—we would've had more dependencies if we had a package manager).

Why not rewrite

• It takes time and effort that could have been spent on fixing bugs and implementing features. Counterargument: C++ might limit or undermine our ability to fix and improve.
• C++ is supported on pretty much any platform. We will definitely stop supporting some platforms if we choose another language.
• C++ is one of the most popular languages, thus the pool of potential contributors is large
• TUI applications never resulted in a rich ecosystem of frameworks. ncurses is pretty much all we have. There is nothing better, and there are no alternatives. STFL hides this from us at the moment, but as we move to another language, STFL will definitely be gone, and we'll have to confront horrors of zillion terminal emulators, each with its own warts. We will definitely stop supporting some platforms if we choose another language.

Picking a language

Most requirements for a replacement language obviously and directly follow from "Why rewrite" points, so I won't list them here. Additionally, I would like that language to:

• compile to native binaries, so we stay in the same niche. Canto already covers "newsreaders written in scripting languages" niche; virtual machines (JVM, .Net) just don't interest me;
• have a way to interoperate with C++, so that the rewrite can proceed gradually.

Haskell

Been toying with it for, um, 7 years now.

Pros:

• more expressiveness than we could possibly use
• uses garbage collection
• purity and controlled side effects give strong memory safety guarantees

Cons:

• small number of potential contributors
• like in C++, have to choose a subset of language that we're going to use, and stick to it
• overall, the language is so different from mainstream that it'll be nigh to impossible to lure a passerby to contribute something

Rust

I read "The Rust Programming Book", contributed to coreutils, wrote a small project.

Pros:

• strong safety guarantees
• relatively similar to C and C++

Cons:

• the killer feature (ownership model) will deter potential contributors
• looks like there are a lot of ritual even around the simplest of operations. String handling is a bright example (but need to take a look at the results of 2017 ergonomics improvements)

Go

Wrote a simple short program about a year ago.

Pros:

• impressive garbage collector with very brief pauses

Cons:

• lack of generics
• weak protection against data races and such

[carno]

Regarding current build system challenges, did You consider using CMake or Meson?

[polyzen]

I often play with the idea of rewriting this in Rust (though lack the chops to do so) ^^. Was toying with rewrite names just yesterday, and @gnubeest bested me with "oxynews".

[ForNeVeR]

You should also take into a consideration that you'll need to interoperate with your existing code, i.e. cheap interop with your C++ code will be required. I doubt that it'll be possible to rewrite any project in a new language at once, so you'll need a gradual approach when old and new code will coexist in the project for some time.

[ForNeVeR]

Also, I have some complaints about your arguments in the section "Why rewrite".

It looks like two of your arguments are directed not to the language itself, but to the ecosystem: you have complaints about your current build system and (lack of?) package management system. And, guess what, there're modern open-source build systems and package management systems for C++ (I won't advertise any of them here, but I could give an advice if you need).

And the first argument ("too low-level") isn't about the language, it's about your code. C++ allows to write high-level applications, it's just the current newsboat code base is written in a low-level manner.

I think that with such an argument set, any rewrite will solve the stated problems, even if you're rewriting the program in the same language. And, possibly, that approach will be the most appropriate one, because it won't create any interoperability troubles and will allow to port the code to the new base gradually.

Note: I'm not telling whether you should or should not rewrite the project in another language, I'm just commenting on your arguments.

[Minoru]

@carno, I did try CMake, see #14 for results. I'll add this to the top post.

I never heard about Meson, thanks for mentioning it. Its design goals seem very fitting, but I'm put off by dependency on Python. Newsboat used to depend on Perl, which led to some complaints from people who tried to cross-compile it. I'm pretty sure Python will get us the same. But I'll try it out the next time I get fed up with status quo :)

[Minoru]

@polyzen, I myself am considering only gradual rewrite that doesn't change the name. Newsboat contains quite a number of little gnarly details, I'd rather port them than try to recreate them from ground up.

OTOH, if you do start a rewrite, I might just steal some code from you (license permitting)… Looks like it's in my best interests not to dissuade you :D

[Minoru]

@ForNeVeR, good point re C++ interop—I'll mention it explicitly in the requirements list. Thanks!

"Why rewrite" is indeed quite weak, mainly because I don't see a single core problem. As I said, it's a death by a thousand cuts.

Re build systems: apart from CMake, I also looked at waf, which didn't impress me at all. IIRC it didn't offer much improvement over Makefiles, so I gave up without even pushing the code to the repo. Anything else I should try (not counting Meson that's been mentioned above)?

I think I'll do some research on package managers now. I've heard of Conan, but I never tried it. I'm a bit put off by necessity to choose, though. With more modern languages, I just get a package manager and can be sure any dependency can be installed through it; not so with C++, apparently. I admit that's more of an emotion than a rational response :)

Re "C++ allows to write high-level applications", in short I want a language that not just allows, but also helps.

I'm using boyscout rule to gradually drag the code towards more high-level C++, but it's too slow to my taste. It's 2017 and we're still using C++11 because that's what majority of compilers support. We could probably move to C++14 already, but it was a minor release from my POV, so no hurry. A lot of useful stuff isn't even in C++17, so won't become available till 2020. And then it'll be what, 3 to 5 years to get the compilers? So by 2023-2025, I probably will be able to use cutting-edge technologies like modules and such. But I'll still have to be aware of weak typing, still put my signatures in both .h and .cpp, and so on.

And then some things will never be fixed because the Committee doesn't like to break existing code. I am just re-evaluating how relevant this stance is to me.

[Minoru]

A few more package managers to look at: Build2, Conan, Mason, pacm (seems abandoned), Yotta.

[kmws]

I agree with points of @ForNeVeR.

too low-level for the task at hand

I disagree, but I think I know what you mean by writing so.

It feels kind of low-level in some parts of the code, yes, but it doesn't have to be like that. I saw programs written in C that read like some 'high' level language and you didn't feel encumbered by lack of abstractions (built into language) at all. I believe that it comes more from programming skill level (and I don't mean by saying so that I have achieved some high level!) and experience rather than by chosen language. Yes, C++ allows you to fiddle with bits, but it also allows you to build many high-level concepts (generating some problems in the process, for which it is hated by many). That's why it's sometimes called "a flexible shotgun".

lacks convenient build system
lacks convenient package management

It's not a language problem per se, but I understand that it is a strong point of concern for you as a maintainer.

I'm a little scared by Makefiles myself, but on the other hand they are venerable in the UNIX world and many people had to fight with them (for a long time), so they are pretty stable and many people can offer a hand. For an open-source project I guess it's not so bad.

I also have to say that I don't believe in silver bullets, and the package managers from the languages you mentioned probably have problems on their own (and maybe they are a moving target in terms of rules/stability). The bright side might be that they have more active/welcoming community that can offer help.

As for the languages you mentioned, I wouldn't be able to help in any of them.

Haskell

I sadly don't know it yet. It's definitely on my to-meet list, but not now, and even if, I don't think I will be able to express myself freely in it for some time. I think you will probably loose most of C++ devs (maybe even start from scratch?), but I don't know how about attracting new ones interested in Haskell? Hard to tell. Change of programming paradigm seems like a big deal to me.

On the other hand, you seem pretty in love with it, and your happiness as a maintainer should be a top priority for all of us. :)

Rust

I haven't tried it. I didn't jump on the "rewrite World in Rust" bandwagon (yet). It's just my opinion, but it also seem to come with a specific political baggage. This can paradoxically broaden the contribution range.

Go

Interesting language, also want to try it out, but I have the same problems as with Haskell. Most of the folks writing in it are developing some kind of "servlets", I don't know how it is like in utility apps.

I would like to say that there is a D programming language which seemed interesting, but I don't think it has any big-player backup like the others do, so contribution factor will probably be very low (in the near future). Mentioning it more for the love for the concepts than anything else.

Rewriting is a big deal that can kill a project. I would vote for refactoring the most offending (in terms of code cleanliness/libraries used) parts of Newsboat and put it in C++ 11/14 range which is pretty nice in terms of standard library. The more code is easier to maintain/extend, the more fun it is to work on it (IMO), which can lead to more contributions and easier development.

[Minoru]

Thank you, @kmws! I've heard of D before, but haven't tried it yet. I'll mull over your technical points for a while, but there's one thing I want to point out right now:

On the other hand, you seem pretty in love with [Haskell], and your happiness as a maintainer should be a top priority for all of us. :)

That would be true if I wanted to be the sole developer, but I don't. So it would be nice if more people were happy, and thus felt motivated to contribute.

[Minoru]

I've had a few conversations about this with people both inside and outside of the community, and finally realized that the crux of the problem isn't C++, it's me. In short, I'm quite fed up with C++, and don't want to deal with this frustration both at work and at home. Since it's much easier to change a hobby project than a commercial product, I'm changing Newsboat instead of the thing I'm building at work. Switching jobs is out of the question right now.

Why do I feel entitled to rewrite the whole project in a new language just because I can't deal with the current one? Actually, I don't. I just truly believe this is the most reasonable choice I can make. My thinking is outlined below. "Problem solved" means I don't have to deal with C++ anymore, and I as a user have a newsreader I enjoy using.

1. If I continue as-is, I'll either:
   1. change my attitude for whatever reason (mature a bit, maybe?) ⇒ problem solved;
   2. get to a point where I just don't want to work on Newsboat anymore, so I quit, and then:
      1. someone picks up where I left off ⇒ problem solved;
      2. no-one cares, Newsboat dies ⇒ I don't have a newsreader anymore, very upset, pick it up again and start rewriting fearlessly because now I know for sure it's either me or no-one;
2. I start the rewrite now, and the world's reaction is either:
   1. positive ⇒ problem solved;
   2. indifferent ⇒ same as positive to me ⇒ problem solved;
   3. negative (contributors leaving expressing strong disagreement, upset users open angry issues and write blog posts, etc.), and then:
      1. no-one cares enough to fork ⇒ I'm very upset because some people don't like me now and I have proof of no-one caring about this; but I still have my newsreader ⇒ problem solved;
      2. someone forks Newsboat to continue development in C++. That's a healthy reaction IMHO, and I'll race that new maintainer for a while, just in case they aren't serious. Then either:
         1. the fork gets to a point where I want to use it ⇒ i retire or pass Newsboat on and switch to the fork myself ⇒ problem solved;
         2. the fork lives on but doesn't interest me-as-a-user ⇒ I keep maintaining Newsboat ⇒ problem solved;
         3. the fork dies ⇒ I'm sad because I hoped for options a. or b. but still ⇒ problem solved.

I can't count on 1.I. Thus, I'm left with the choice between 1.II.b (burn out then rewrite) and 2 (rewrite and maybe get negative feedback), of which the second seem the most reasonable.

Where does this leave all the current contributors? I would prefer to keep working with each and every one of you, but I understand not everyone wants to pick up a new language. I'm truly sorry to exclude you so, but I just can't continue with C++.

Getting back to the technical side of things: I skimmed first few chapters of "The D Programming Language" by Alexandrescu and just googled around a bit. I have to admit C++ interop tempted me, and the language overall didn't seem too off-putting. I disagreed with not enforcing curly braces after if, and mutability by default, but D is still a language a could program in. Yet it has weaker memory guarantees than Rust or Haskell, and seem to try to lure me with promises of familiarity and comfort, which make me suspicious—language design has to deal with a lot of little controversies, and this "comfy" attitude will result in picking weaker but more popular defaults (like no block enforcement and mutability I mentioned above).

I chose Rust.

I will be opening more refactoring issues to further modularize our current codebase, and in time, also issues for rewriting parts in Rust.

[kmws]

A hard decision for you, for sure. As I stated above, if you're not happy as a maintainer, you will just burn out sooner or later resulting in badness for all of us.

On the other hand, that's a sad decision for me, because I can't join this boat (:-)) and contribute further, though I will surely be using Newsboat in the meantime, because it's just cool.

[Minoru]

Thanks!

On the other hand, that's a sad decision for me, because I can't join this boat (:-)) and contribute further

Well, if you're into refactoring and figuring out other's code, there's still plenty of work to do. Documentation needs attention, too—it has been gradually extended for years, and might benefit from restructuring and streamlining. You're still welcome!

Edit: LOL, accidentally added a reaction to my own comment instead of yours.

[bartlibert]

Hmm, might be a reason to start learning rust as I have meant to for quite a while :) (Not that I'll have a lot of time to contribute huge chunks of code I guess)

[chiraag-nataraj]

I vote for Haskell, if only because it's the only language out of the three that I actually know 😉 😂

[kpcyrd]

This thread was linked here: https://groups.google.com/forum/#!topic/newsboat/y3cylBEiDhI

Are there any other resources, like a meta thread for the port? For example a discussion how the transition phase would look like, especially if a release is published during that time. I'm involved with rust on debian so I might be able to help with some of the issues.