The Qodem Project Work Log
June 11, 2017
Thanks to Ryan de Laplante, Qodem can now dial actual BBSes! I needed to do two major things to get it working, and afterwards I find a little more still. The old buoy flush_serial_port() code is still causing lost bytes immediately after connection, and that has to go. flush_serial_port() made a lot of sense for the embedded buoys, but it doesn't belong here at all.
The final release date is now set: June 18, 2017. This is the final week.
June 2, 2017
I have been tackling a dialup failure a user reported, using a null modem cable to simulate both ends of the link. I discovered that close() on a serial port could fail to flush the buffer, so added tcdrain() to qodem_write() when sync is true to cover that case. It was needed because it appears that the modem the user is using is either slow to assert or doesn't use at all DCD, so I added a new option ignore_dcd and needed to make hangup_modem() rely on the hangup string (+++ATH0) rather than just drop DTR.
Respun all the builds, and then noticed that I had the same %s baud crash in host mode connection string, so have to do it all over again. Sigh...
May 27, 2017
I've got the release build VMs up and running again. My main laptop can build everything except the Mac binary, but I've got a new Mac Mini (Sierra) to do that now. I need to check the man page and help text, update the BBS list, and sweep the README and index.html. Might get some of that wrapped tomorrow.
I've got 38 days until July 4, but need to release before June 22 so about 25 days left on the clock to 1.0.0. So I will get the latest BBS list and spin the final build on Windows first, and then keep going down the TGR list. I probably won't hit every item on that list, but even so this will be the most stable release since 0.3.2.
And one that I will be content being the last release for a very long while. It is now time to put in writing my feelings on a few matters...
Qodem has taken a very long effort, and I feel that I am reaching the finish line just in time for the last staff member to be leaving the stadium and turning out the lights. Or perhaps one could say I am putting the finishing touches on one of the coolest transoceanic sailing ships in the world, at a time when no one has enough vacation time from work to actually travel in it.
The programming world has become all about Optional, functional programming, and HTML/CSS/JS on the backend. The Internet of Things (Internet of Shit) is taking over seemingly everywhere. Just today I had to spend more money for a house thermostat that didn't require wifi or a twelve-page manual to program it. On the Linux side, systemd ... well, I didn't have any real problems with init; nor did I have a need for PulseAudio so that videophone calls could seamlessly transition from speaker to USB or bluetooth headphones, I was doing alright with alsa. In fact, I could be pretty content with KDE 3.x, Debian Sarge, and Java 1.7. Or on the other side Windows 2000, Office XP, and Firefox 3.x.
Modern computing isn't what I like anymore. I don't want to HAVE to be connected to the Internet or my game won't run, and I don't want my motherboard to REQUIRE some unauditable "security processor" to have backdoor access to everything behind my operating system's back when I AM connected to the Internet. I don't want my IDE to have to go online to pull a zillion dependencies just to get Hello World running on whatever today's latest fashionable GUI framework. And flat icon themes, just fucking yuck.
Last year I moved houses, and I took the unusual step of not moving my broadband service with me. My wife has one smartphone which we share for email, weather, and light browsing; I have a dumbphone for texting and voice calls. I do my basic news headlines from work, and we go to the local library a few hours each week to do bills and (for me) sync with github. It all reminds me more than a little of BBS life. I PLAN my time to be online now, it doesn't follow me everywhere. I notice all the other things that are not being done while I am online, because online means not at home now and things like dishes and yardwork and reading physical books is not happening when one is clicking away at web sites. Sometimes I really look forward to a few hours at the library, accessing the sites my work filters off. (Note to industrial manufacturing companies stuck on the Microsoft stack: filtering out open source sites is NOT helping you be more competitive.)
But unlike the BBS era, the world will never come to understand what I like about MY stuff. They are headed off to the singularity/cyberpunk futures of Rainbows End and Idoru: near-total VR immersion and augmented reality. What I want -- computers that work as I (and no one else) tell them to, computers that make ME a bigger thinking entity, programming languages that favor stability and teachability to others over syntax fashions and IDE glitz -- will never become the focus again. There is no money in it. And there never were, and never will be enough people with the capability to do this work that will pay for the infrastructure needed to sustain it. Why would any business approve an engineer's request for a $10k POWER system that behaves as a computer should when there are $1k ARM/X86 spy-boxes for sale in every grocery and office supply store? The money instead is in turning computers into telescreens, windows for other entities to peer into one's life to slice and dice and carefully measure the masses of humanity. To find out their dreams in order to manipulate them to support the existing social order, to discover their talents and passions in order to nudge them into more profitable avenues, and as a last resort to collect blackmail should they decide to enter politics. The Internet pioneers of the IETF, IANA, and everybody else thought they were creating a fertile breeding ground for anyone to become the next Craigslist / Reddit / Amazon / Google / Netflix, but what they have actually brought about is the Great Firewall and the Snowden revelations.
I am glad that Qodem has reached this point, that it is in my mind "complete." But I am also sad that in the time it took to get here the world abandoned the general-purpose computing concept which tools like Qodem are intended to support.
After Qodem is released, I will be taking a step back from programming in general. Some things I've learned will still be published, someday: Jermit for the file transfer protocols, Jexer for Xterm- and Unicode-isms. Bug fixes will be worked on, but at a more sedate pace.
March 11, 2017
The Great Regression (TGR) finally begins. Only about 8 years after I first put together the list, and about 40% of the total code lines ago.
This one is going to be quite a bit different that anything else Qodem has undergone. If I'm lucky, the only thing that will be updating between git commits is this log and the TODO list, showing more tests passing. If I'm less lucky, then I will encounter bugs that have to be addressed before the final 1.0.0 release.
Release target is 47 days away, there are about 150 items on the test checklist. Some items like file transfers will get knocked down quickly: I am confident that they work, and it will be just a couple hours to try all of them, and 10% of the total list is done. But some like the Phonebook revise screen are several hours just for one line in the checklist. I can't possibly test all of these combinations against all the supported platforms, so I will divide these up but put the majority of the effort on Win32. Serial ports and scripts will need to be tested on both platforms though.
Along the way I also need to slow down. Part of me wants to just #define RELEASE, spin the builds, and off we go, and then I can put Qodem down for some months (or years if there are no real bugs). But this particular release is a bit more significant than the others, and I need to really put it through its paces. This will be a slow burn, yet needs to be put on the back burner too because it is about to be spring in my new house and I need to get some real life going. If it turns out that I miss the April 27 date, then the next date out will be July 4.
I finally found the X11 selection bug that has been killing me for the last two years. Somehow Xlib is sending a mouse event with button number 0x7F00, which crashed a dereference (button should be 0-5). Adding the check seems to fix things. I'm going to get this pushed out tomorrow as the new-and-improved 1.0.0.
March 10, 2017
Borland has some serious problems with colors when 64-bit chtype is being used. I don't have much time or interest in figuring it out, because I am more surprised that it worked in the first place. I added CHTYPE_32 to the win32a and qodem builds and all the problems suddenly vanish.
Fixed the Windows blink to match the behavior of X11.
March 9, 2017
My Win32 double-width support introduced a memory smash, got that fixed now I hope. Also got a color icon for the X11 version.
March 5, 2017
I have started using Qodem as my daily driver under X11 again. Naturally I am finding lots of small glitchy things again: when my default color has bold then LOTS of other stuff was bold too; Jexer was relying on Xterm's erroneous behavior of bold'ing the background color for its color picker; Shift-Tab wasn't being picked up (x11.c didn't have the keysym in its table); and I noticed that scrollback page-up/down was actually skipping a row (and has been doing so for probably 12 years now). So let's get those fixed now that I'm seeing them.
It is weird though, seeing Linux for real using the DOS color palette. That bold + reverse behavior of Xterm has been with me so long that seeing CGA bold behavior is really driving home the "vintage" feel.
Also, Qodem has formally entered freeze for 1.0.0. It's been a long 14 years, but we are finally just about here now.
2186 5925 55727 source/ansi.c 81 357 2454 source/ansi.h 510 1635 13740 source/atascii.c 68 266 1965 source/atascii.h 931 2337 24274 source/avatar.c 56 186 1498 source/avatar.h 2189 10906 92406 source/codepage.c 302 985 8063 source/codepage.h 1761 12781 87397 source/colors.c 282 852 6396 source/colors.h 541 1711 12417 source/common.c 337 1375 9532 source/common.h 80 392 2317 source/config.h 3918 10799 127105 source/console.c 179 664 4661 source/console.h 1006 2784 28552 source/dialer.c 132 385 2880 source/dialer.h 1424 3858 37112 source/emulation.c 222 856 6681 source/emulation.h 594 1862 16176 source/field.c 276 851 5934 source/field.h 3948 10729 130751 source/forms.c 252 1409 9131 source/forms.h 2891 13407 107235 source/help.c 71 242 2095 source/help.h 2914 7841 80967 source/host.c 125 439 3177 source/host.h 2422 8134 72032 source/input.c 274 910 8069 source/input.h 6719 19954 195548 source/kermit.c 84 381 2659 source/kermit.h 3833 10079 113312 source/keyboard.c 80 275 2152 source/keyboard.h 2805 7515 81521 source/modem.c 291 973 7384 source/modem.h 1431 4604 42483 source/music.c 99 381 2805 source/music.h 5055 15911 146646 source/netclient.c 334 1469 8964 source/netclient.h 2184 8771 77352 source/options.c 255 750 7223 source/options.h 1271 4444 39157 source/petscii.c 77 318 2290 source/petscii.h 8600 23536 282950 source/phonebook.c 230 774 6132 source/phonebook.h 2343 6660 74078 source/protocols.c 290 1044 8075 source/protocols.h 74 290 2151 source/qcurses.h 3776 11952 119319 source/qodem.c 783 2744 19188 source/qodem.h 1552 6487 47347 source/screen.c 575 3120 19515 source/screen.h 1748 4970 49830 source/script.c 176 606 4308 source/script.h 3082 9978 94928 source/scrollback.c 406 1705 11359 source/scrollback.h 456 968 11711 source/states.c 166 540 4573 source/states.h 2401 7365 71622 source/translate.c 150 580 4411 source/translate.h 7770 22233 235670 source/vt100.c 153 650 4532 source/vt100.h 887 2124 21305 source/vt52.c 73 282 2060 source/vt52.h 3104 9913 88958 source/xmodem.c 114 554 3908 source/xmodem.h 5611 16942 164502 source/zmodem.c 87 402 2824 source/zmodem.h 99097 317122 3045496 total
March 3, 2017
We are down to the last feature before 1.0.0: Win32 double width support. And I even have several hours available this weekend between coughing fits to make the attempt.
... later ...
We have double-width in Win32, and it was amazingly easier than X11 thanks to StretchBlt().
February 26, 2017
Qodem has many more options for portability now. --config lets you bypass all of the standard qodemrc locations and just load a specific configuration file. --create-config can put a qodemrc wherever you want. --dotqodem-dir lets you change where ~/.qodem is. And --read-only disables scripts, saves, downloads, and similar writes. Between all of these, one can easily create a shell/batch that runs qodem to either a) be a passive client to other systems, or b) run entirely off a USB stick.
February 23, 2017
I ran across a version of Qmodem I have amazingly enough never seen before: Qmodem 4.5 Lite. It has a compile date of 01/01/1992, as compared to Qmodem 5.0's date of 01/06/1992. It appears to be almost identical to 5.0, but without script support or the Alt-M mailer command. But nicely enough it also doesn't have an installer: drop the files into a directory and off you go. I must be getting old, because at age 15 I would never have wanted the "stripped down" version, yet here 25 (!) years later I'm thinking 4.5 Lite might be the single best version in the collection.
February 18, 2017
We are closing in on PETSCII. It is fascinating reading about and playing with C64 via VICE. I often wish I was just a few years older and could have done the 8-bit computer thing as a teenager. (Even better would have been graduating with my CompSci degree in the early 1990's rather than 2000, my whole life would have so much better financially. Ah well, at least I won't be taken by surprise by the re-rise of the new feudalism.) This afternoon I'll try a couple C64 BBSes. If it goes well then I can move over to ATASCII.
February 17, 2017
I asked the great Internet what PETSCII font makes the most sense, and the answer is C64 Pro Mono from STYLE. So I will be using that as my standard baseline. Interestingly though, it appears that xterm recognizes that font as a double-width font anyway, and fails to "double the double", opting instead to put spaces in between each glyph -- the same thing Qodem does when it does not assume its terminal can do double-width. So my PETSCII emulation needs to support both 40-column-with-a-good-C64-font (which means NOT setting double-width) and 40-column-with-a-narrow-non-C64-font (which means setting double-width). Fun fun.
February 12, 2017
Double-width/height appear to be working on X11 now. I had to add a cache for performance to not totally suck, but it is there now and will be a model for the Win32 version.
November 10, 2016
Whew, there has been a decently large code sweep. The control keys keyboard handling now passes control characters down to the emulations. 40-column support is stubbed in the X11 backend, it has more glitches than working parts but it is a start. Gonna pack this into a commit and then push it up.
November 6, 2016
Tim has been playing with the code and found several interesting items to be looked at.
First, SGR 38/48 are causing crappy output on XTERM/X_UTF8 due the awful broken-ECMA48-ness of those sequences. I've opted to recognize them for XTERM but silently discard them because I cannot easily do direct RGB on all my backends. Keeping a faithful VT100/VT220 however requires that SGR 38/48 result in blinking off-color text.
Second, Tim put together an ATASCII branch and exposed some issues in my keyboard handling of backspace and enter. The general problem is handling of control characters that cannot be directly sent to the other side as is. He chose to remap C_CR to Q_KEY_ENTER and ^H to Q_KEY_BACKSPACE in keyboard_handler(), and removed the Q_KEY_ENTER cases in terminfo_keystroke() and bound_keyboard_keystroke(). It's an interesting find, which will help on PETSCII too. C_CR can become Q_KEY_ENTER pretty easily (after changing notify_prompt_form_long() to match '\r' with Q_KEY_ENTER), but ^H and Q_KEY_BACKSPACE will need some deeper thought. (These remappings should also be done in qodem_win_getch(), not keyboard_handler().)
On ^H, we have several competing issues: backspace toggle (Alt-2); VT220/Linux/xterm mapping the actual backspace key to DEL which I then remap to KEY_BACKSPACE (and I just noticed the VT220 terminfo is wrong, it has kbs=^H when the VT220 manual defines that key as DEL); and the occasional need to send ^H to the remote side (which can also be accomplished with Alt Key Code). Looking closer, I am already mapping ^H to KEY_BACKSPACE on PDcurses because it never uses KEY_BACKSPACE. So the best thing here might be just to shove everything into KEY_BACKSPACE and eliminate all use of 0x08 in the forms code.
To really finish this up though, I also need to look at 0x09 (TAB) and 0x1B (ESCAPE). Those are being passed through directly too, and KEY_ESCAPE is actually a bit awful in that the emulations are looking at it rather than C_ESC.
...It's later, and I think most of the keyboard handling is better. Control characters "belong" to the emulations, except for any embedded in keyboard macros (which will transmit raw to the other side). I found a couple assertion fails along the way to fix.
Gonna test this a bit more on some systems but it is close to being ready for the rest of PETSCII and ATASCII. When I get around to PETSCII/ATASCII limitations in the README / help I will need to note that carat control characters (e.g. keyboard macros, modem, split screen) are sent raw to the other side, not processed through the emulation.
October 28, 2016
I believe I found the hang on exit and have that fixed. I also noticed that SDL_CloseAudio() will sometimes hang too, I think it was bumping heads with my background movie audio because it went away afterwards. Weird.
I have stubs in place now for PETSCII as an emulation. On ncurses/xterm it uses double-width 40 columns, which is very slick. I discovered that my xterm double-column trick was screwing up non-console screens (input boxes, switching states, etc.) so added logic to clear the double-width for those transitions.
Next up is taking the weekend off, and then looking into double-width on Win32/X11. I would REALLY like to get PETSCII looking nice on Win32, I think it would be cool as hell.
October 27, 2016
It's been a productive day so far. My Win32 and Linux dev environments are back up and running. I found some crazy behavior between Qodem and inetutils-telnetd where telnetd was switching back to ASCII and then also sending a bunch of C0 where no one else does: NUL's and SOH (0x01) in places. Qodem was definitely doing the wrong thing with turning every CR into CRLF in post_keystroke() (but only for ASCII telnet), so I got that fixed. I also ran it against clang, eliminating most of the compile issues seen on recent OSX (issue #39).
The C64 / PETSCII request (#41) sounds like it would be really fun to do. I've grabbed a copy of CGTerm and other reference materials, and will be poking away at it off and on. It will need to be its own emulation (it's far more than a codepage), and I haven't added one of those in a long time so that will be nice too.
Time to wrap today's work and get this pushed out.
October 26, 2016
Back in business! I am in the new house and out of the old storage units. I still don't have my workstation set up, but have found time to visit the library to borrow their Internet and keep moving along.
Today is a rather neat day. Qodem just merged its first pull request from Github. :-) Thank you to Tim Hentenaar for your bug fixes.
September 16, 2016
I am still alive, and this project is not dead. My summer has been slammed with moving to a new house and several overtime work projects, leaving this one no extra time. I am hoping that I can put more time into Qodem in about another two months. Until then, I am still looking at issues and do have plans to clear them all before releasing 1.0.
April 30, 2016
I merged current 1.0.0-dev back into master yesterday and put some dev snapshots (source and Windows installer) on the downloads page. Still working on the 1.0.0-dev branch for a few more items and will re-sync to master about every week or two.
My favorite new feature is "xterm mode", activated by the --xterm command line flag. It makes Qodem startup and behave very similarly to an actual X terminal: no banner, no phonebook, no networking, no serial port: just a local shell and then Qodem exits when the shell does. Yet we still have scrollback, capture, zmodem/kermit, keyboard macros, and all the other goodies. Combined with support for window title changes and Shift-PgUp entering scrollback, and qodem-x11 will soon be my new permanent X terminal. I can finally eat my own dog food 100% of the time. Scary, but exciting. (I'm already noticing some latency that will be driving me bananas. So that will get fixed soon too.)
In more personal matters, we are cancelling our home broadband Internet connection soon. I intend to spend a good deal of time this summer outside in the yard and further about town. I can still do Qodem development offline, and I can check github and personal email during my day job, but commits will probably come in about every week or two.
April 28, 2016
Well, that's all the bugs I currently know about cleared off the queue. Still have those two TODOs in the code, but we are in a nice place. ncurses keyboard handling is much more sophisticated now, parsing modifier xterm sequences and also passing them through XTERM/X_UTF8 to the other side. The host mode bugs are basically ironed out, though you can still get a message messed up if chat mode is used, however now you can re-view the message before saving it. If miniupnpc is already on the system then autoconf has a decent change of finding and using it.
I just need to keep banging on it for another couple months and then release 1.0.0 in July.
April 24, 2016
I've gotten a few bug reports come in, which is really nice. Bumping the dev version number to 1.0.0 (woo!) now. I've decided to disable cryptlib by default in configure.
April 21, 2016
I am down to two remaining TODOs in the code: verifying that the comm timeouts in Windows serial port are OK at various baud rates, and implementing a secondary strategy to map Unicode back to 8-bit. DMG is building for OSX, Qodem does run in Terminal.app (a little ugly, but better than nothing), cleaned up the ChangeLog, and moved the last two host mode bugs to the git tracker.
Some final testing for a few days, and then I will start spinning builds. 1.0beta is coming out within a week.
I lied, in a good way. 1.0beta is releasing TONIGHT!
For nostalgia's sake (last done in May 2004), here is the 'wc' output:
2168 5894 55343 ansi.c 81 357 2454 ansi.h 920 2308 23849 avatar.c 56 186 1498 avatar.h 2173 10876 92040 codepage.c 292 946 7715 codepage.h 1742 12687 87996 colors.c 282 852 6396 colors.h 522 1664 12060 common.c 336 1373 9494 common.h 3592 9752 114804 console.c 179 664 4661 console.h 956 2661 26958 dialer.c 132 385 2880 dialer.h 1313 3522 33423 emulation.c 220 846 6569 emulation.h 594 1866 16201 field.c 276 851 5934 field.h 3952 10719 130633 forms.c 252 1409 9131 forms.h 2825 13064 104528 help.c 71 242 2095 help.h 2813 7559 78087 host.c 125 439 3177 host.h 1866 5438 51069 input.c 266 886 7835 input.h 6704 19900 194876 kermit.c 84 381 2659 kermit.h 3798 9966 111823 keyboard.c 80 275 2152 keyboard.h 2803 7512 81465 modem.c 291 973 7384 modem.h 1421 4569 42226 music.c 99 381 2805 music.h 4971 15596 145009 netclient.c 329 1456 8877 netclient.h 2082 8042 71440 options.c 231 697 6643 options.h 8473 23026 278234 phonebook.c 229 772 6105 phonebook.h 2343 6660 74073 protocols.c 290 1044 8075 protocols.h 74 290 2151 qcurses.h 3205 10156 103155 qodem.c 699 2464 17267 qodem.h 1346 5921 42253 screen.c 571 3080 19295 screen.h 1724 4906 49173 script.c 176 606 4308 script.h 2969 9604 91148 scrollback.c 406 1705 11359 scrollback.h 450 961 11610 states.c 166 540 4573 states.h 2373 7235 70606 translate.c 150 580 4411 translate.h 7479 21150 224054 vt100.c 152 639 4451 vt100.h 885 2121 21297 vt52.c 73 282 2060 vt52.h 3088 9855 88262 xmodem.c 114 554 3908 xmodem.h 5611 16942 164502 zmodem.c 87 402 2824 zmodem.h 94030 298689 2883343 total
April 20, 2016
We are getting veeery close. Updated the BBS list to the current month, fixed more codepage/widechar things, eliminated most of the remaining TODOs (by way of documentation), and updated the man pages and help text.
April 19, 2016
Both translation UI's appear to be complete. Preliminary testing is good, Unicode only in/out and back-converting Unicode to 8-bit are both working.
April 18, 2016
More of the translate system stubbed in. 8-bit translates are working again. Stubs are in place for the Unicode-to-8-bit back-mapping also.
April 9, 2016
I have stubbed in new phonebook fields for 8-bit and Unicode translate tables. Along the way I have also broken existing 8-bit translation: qodem no longer knows what translate.tbl is. But this is a significant start. Next up is making the 8-bit table editor aware of the filename, then putting in something similar to the keyboard switcher: there will be default mappings (which do nothing) and then per-phonebook entry mappings.
After that is the Unicode editor, which I have an idea for. Basically the same editor as the 8-bit, except in 4 columns of 16 rows and using hex notation. Add a couple shortcuts to 'F'ind the key (input) or the value (output) so people don't have to page through 64-1024 pages to find the code of interest, and the editor is mostly done.
Gonna put this in git for now.
March 24, 2016
Just tried the latest cryptlib to see if it fixed the cipher negotiation failure with modern OpenSSH, and it did not. This one will have to wait until after 1.0beta. This royally sucks for the Windows version though. :-(
I fixed the compile errors in miniupnpc, now it is alive again on OSX.
So where I am at right now is that I could conceivably call this 1.0beta today. The new translate table stuff is a good idea, but it could also be put off to 1.1 and few people would be impacted.
I am going to respin the Windows build and take a couple weeks off. This has been a fantastic push and I feel pretty great about where it is. Now I need to give myself some more me time outside work and rest up for the final leg next month.
March 23, 2016
The RPM build is working again. I had to use Fedora 22 rather than 23 due to 23 crashing under VirtualBox (other have seen this too). Worked out pretty well, it was almost longer getting Fedora running than it was testing the two builds.
March 21, 2016
I announced my intention to release 1.0beta to the world a few days ago, and happily enough I got a couple bug reports to immediately work on. I believe both are addressed now, but we will see what the users see on their end.
While investigating IVT (http://home.planet.nl/~ruurdb/IVT.HTM), a terminal with a really impressive vttest score, I noticed that it had a feature to automatically remember passwords on logins. This sounds super convenient, and a real perk for a terminal to do the kinds of things we normally think of in web browsers, but it left me feeling a little weirded out. Which was weird, because I normally see cool features that are harder for me to see how to do and want to try to crack them. But saving passwords kind of freaked me out. Maybe it was also because I was playing with Arch Linux and their shell completions are a lot more that I am used to.
But it led me to a new philosophy statement for Qodem: Qodem does not try to be aware of semantic meaning or context on the wire. The reason I want this is because I don't want Qodem to act "independently" of the user, I don't ever want to watch Qodem do things I'm not expecting it to. (The only exception I make is Zmodem and Kermit autostart, but even these have safeguards against trashing data.) So it's OK for a script to take control -- there is the nice status bar indicator and the split screen for its stderr -- and it's OK for the user to press a function key to automate some things. But it's not OK for Qodem to act in some way other than "be a terminal" to the data it sees from the remote side.
This eliminates three features from the wishlist: autotrigger macros, Intellisense-style completion window, and parsing the current command line to do ... something.
March 18, 2016
Resurrected the debian builds. Still need to test the RPM builds though, not sure if I want to use CentOS, Fedora, or other for that. I also added the Telnet BBS guide to the Windows install, though it only works for the installing user.
I am getting very close to 1.0beta indeed. I feel like I am now gathering all these pieces strewn across thirteen years of work: builds, features, documentation, icons, and ... well, "feelings" I suppose. It's time to abandon features that I never really needed and tie up the remaining loose ends.
1.0beta now has its final deadline: Qodem's 13th birthday, April 27, 2016.
March 17, 2016
I think host mode can answer the phone now. I can't verify this because I don't have a phone line, but the structure is all there and maybe it will work.
March 14, 2016
We are getting somewhere finally. I fixed several minor bugs, including the blinking text not working in Windows bug. The help text title is working, the Windows build is now in Release mode so it doesn't assert a debug warning on ReactOS, I've updated some documentation, and the initial phonebook entries now include several interesting systems.
I discovered that Big-5 encoding can hose the UTF-8 decoder making ptt.cc a no-go for Qodem at the moment. (...sigh...) Well, I can't in good faith release 1.0 without supporting the largest BBS in human history now can I? So basic/obsolete Big5 to Unicode will be coming in.
March 6, 2016
More bug fixes. Now I know why mouse motion events aren't working in PDCurses: turns out PDCurses doesn't support them in the first place. That eliminated two of the Windows issues I was counting (drag to select and no motion). XTERM now clears the screen and does DECRC/SC for alternate screen selection, which is somewhat better but not quite as nice as a true xterm. My list is now down to two things: modem support for host mode, and the new translate tables design. The latter will be saved to do later in April (my wife will be working an evening job so I can dedicate several 4-hour chunks to it). Modem host mode will probably come in sometime in the next couple weeks: I can simulate the modem using my other PC and a null modem cable.
I like yesterday's Windows release so much that I am going to put it out as a "beta preview". (God I sound like Google. Whatever.) This is really a nice spot function wise, just need to keep finding and banging on bugs.
March 4, 2016
The Windows build is starting to feel a lot more solid now. I found a nice font that works now (UniVGA converted to VGAMedium.ttf for use in ReactOS), it looks awesome. Added a quick sync script to do the unix2dos conversions for the README.md, ChangeLog, etc., and a cleaner Windows-only README file at the end of the install. I resynced all the code, recompiled everything on VC6, and immediately ssh'd to a Linux box with CryptLib working and everything. I also finally found the reverse video bug on Borland XTERM emulation: turns out the 16-bit chtype makes an expression like "(q_current_color & ~Q_A_PROTECT)" become 0 (because A_PROTECT is A_NORMAL which is define'd as (chtype)0), which inverts to black-on-black. Something I put in to honor "back color erase" a long time ago. Well, fixed now.
I was also pretty surprised when I finally got into it how generally simple Windows serial port programming is compared to termios. It's a lot less "flexible", but for handling PC-to-modem and PC-to-PC situations it seems more than adequate. There will be some work to test what's there, especially as I don't have a real phone line anymore (though I do have a modem), but it's all stubbed in anyway.
I am so tempted just to call it soon and release 1.0beta, buuuuttt... I need to get the translate tables re-worked, and find and fix a few more of the Windows and host mode bugs. I'm hoping to do it soon though, maybe this July 4th.
Time to commit what I've got and get to bed.
March 3, 2016
I just stubbed in the InnoSetup install. Very quick and easy, I like it.
March 1, 2016
I was able to communicate at 19200 between a Win2k VM and a Linux server using a USB-to-serial adapter on the Windows side and a true serial port on the Linux machine. I proceeded to send a file over Zmodem at about 95% efficiency, using essentially no CPU on the Win2k side. Neat, really neat. :-)
Still have some serial port details to clean up, but this is a rather huge milestone.
September 18, 2015
NO MORE MUCKING WITH CRYPTLIB, YAY!
All of the features I need are in for SSH now.
This is actually a pretty important moment: all of the features that were present before the Great Refactoring are available again: UPnP, SDL, SSH, and X11. But now it all runs on Windows with two different compilers, and on Linux has both the simple Makefile.generic and an autoconf configure script.
Speaking of X11, I really need to deliver a positive shout out to William McBrine. I reported the client hang bug for PDCurses back in 2012, and within a few months there was a fix waiting in the thread (which is now in PDCurses CVS head). It was all of a two-line fix, but now qodem lives again in glorious Terminus font X11 frame. I really really appreciate William's work: thanks very much!
I'll need to take a break for a bit due to real life intruding, but this is a nice place to be indeed. From here on out I can focus more on my own code and less on infrastructure stuff.
September 17, 2015
Just finished putting in known_hosts support. I exposed a bug in cryptlib: when you ask for CRYPT_SESSINFO_SERVER_FINGERPRINT_SHA1 it gives you MD5, but when you set CRYPT_SESSINFO_SERVER_FINGERPRINT_SHA1 to check it later it expects SHA1. Yeah, annoying. I've changed it in session/ssh2_cli.c and it seems to work OK without breaking the host SSH server code.
I am very much looking forward to wrapping up the last couple cryptlib things and then never touching that library again. Of course that's a fantasy: I'll need to retrofit my changes to its new version when it gets around to supporting ED22519, Salsa20, ChaCha20, and Poly1305 to talk to modern OpenSSH servers...sigh...
September 16, 2015
I brought in the old autoconf build, stripped down and cleaned up for the new directory structure. Along the way I had to make a GNUmakefile for cryptlib to wrap its makefile, and a new Makefile for PDCurses. Interestingly, I also finally have the answer for why I was getting the "relocation error" regarding ttytype: SDL pulls in ncurses at link time which has a name collision.
So now I am staring at a bug I never got fixed before: the X11 process part (using libXt) of PDCurses hangs during its setup, so qodem-x11 shows up as a blank white screen. I guess it's time to learn enough Xt now to fix PDCurses.
September 3, 2015
On that last commit a few days ago I reached a major milestone but didn't have time to write on it. Under VC6 and Windows 7 (haven't tested XP), cryptlib appears to be fully functional: ssh client and host are working. This is really big, because cryptlib was (let's be honest) a freaking bear to get even marginally working under Borland, and now that I have a working version I can either figure out how cryptlib broke under Borland or just ship a VC6 binary as the qodem release. I will probably end up using VC6 as the release binary, but I do want to support Borland anyway even if it means that ssh is broken.
I'm stubbing in support for the last third-party piece, miniupnpc. While I'm here we'll update to the latest version 1.9 which has some serious fixes compared to the old 1.4. Once this is working on Windows and Linux I'll be back to my old ways of fixing bugs and testing.
August 26, 2015
I've got a VC6 qodem built with all of the same functionality (theoretically) as the Borland 5.02 version. Along the way I fixed a lot of compiler warnings, mostly these where implicit casts and signed/unsigned things in function signatures.
So now I have cryptlib running under its native VC6 compiler, and it's still broken. This time it doesn't look like it's corrupting memory though, it is providing the same error message for both client and server: -11 "Network subsystem not available". For the server it doesn this on setting the network socket, and for the client it does it when activating the session. Yay for more consistent behavior.
Alright, time to pack it in for a bit and get to bed.
August 22, 2015
Just a short update tonight, I'm pretty beat. I've been beating my head up on cryptlib under Borland 5.02. Let's just say that cryptlib is excessively difficult to debug when it is having trouble, due to its kernel dispatch type design: you have to run over and over again through the krnlSendMessage() framework just to find the right spot in the backend that is barfing, and then continually recompile the entire project to see what USE_ flag was missing.
I am now down to the client is sort of working but I don't know why, because it is breaking Borland's tracing and breakpoints as soon as I activate the SSH session. There are other strange behaviors: the entire app locks if I redirect stdout or stderr to a file, my DLOG() stops emitting output (through the program keeps running) after the activated session, and the server socket behaves as though it is non-blocking despite me never calling ioctlsocket() to make it non-blocking (and the listening socket is also non-blocking). So I am left believing that I have left a memory corruption bug in it somewhere. Time for the next turn: getting it running under Visual C++ 6 so that I can compile the cryptlib unit tests and see if the small changes I have made to cryptlib are failing any of its tests. If they don't fail, then I go further and just start debugging in VC until I see what it is that is different between VC and Borland.
But Borland C has proven that cryptlib is possible, and despite all the pain of my debugging I sort of have a better idea how cryptlib is put together now. If I do manage to get this working then I will submit a patch back to Gutmann. (I also have some ideas on how to expose the SSH terminal features, and maybe that will be generalizable enough to be worth doing to mainline cryptlib, we will see.)
So I'm off to commit a qodem that barely compiles in Visual C++ 6. Fortunately it caught a lot of my C89 errors which will make it easier to run on other older platforms, yay.
August 10, 2015
I think I have settled onto the Unicode-vs-8-bit design issues in my head now. I wanted to think about mapping keyboard wchar_t's to 8-bit codepage values but couldn't get past the translate tables behavior. The way they work just doesn't make sense in the bigger picture, and in fact the Qmodem(tm) translate tables feature seems like it probably solved one very specific problem but was just left in place rather than properly "finished", with these major problems:
It is a global setting rather than a per-dial-entry setting. If one set it up for say EBCDIC then you would need to have a whole separate Qmodem(tm) install just for those EBCDIC systems.
It is unclear what program states it is supposed to operate on. ASCII transfers have an option to use it or not, but not the other protocols. Again, this seems like a one-off "translate from EBCDIC to ASCII for text only" type feature.
Here is what I think needs to happen: the translate tables function needs to be "finished" and expanded to make sense in a Unicode world too. I see the need for two kinds of translate tables: 8-bit that operates on the bytes sent to and from the wire, and Unicode that operates on the code points sent to the screen and received from the keyboard. Add a few special cases to handle the non-terminal-mode states (host mode, downloads, UI screens) plus DEBUG emulation and this can all fit together neatly. Core philosophy statements:
We do not record in capture file or user data files the result of a destructive codeset transformation. So: raw capture records the bytes before the 8-bit input translate table is applied, normal/text capture records the code points before the input Unicode translate table is applied, and data files record the code points from the keyboard before the output Unicode translate table is applied. The only places where destructive transformations may occur are in the scrollback and screen output functions. This also means that scrollback capture and screen dump are permitted to have different code points than capture file.
The translate tables are optional, and operate as a per-dial-entry function. This will mean a new phonebook entry field.
If a Unicode code point does not have an 8-bit character to map to, the '?' will be used in its place. This is consistent with Windows OS behavior and browser conventions. Due to #1 though the user files (phonebook, macros) will still have the original Unicode data, and selecting a different codepage might make it show up. In terminal mode this means that '?' will be sent to the remote side for un-mappable code points.
DEBUG emulation targets what goes on the wire. Unicode from the keyboard will be encoded as UTF-8, but show up in the hex display as the 2-6 bytes being sent across the wire (using the global default codepage) rather than a wchar_t. This will be the default behavior, as UTF-8 is the future. However, I suppose I will also provide an option to have DEBUG encode to the 8-bit codepage, with the same caveat that un-mappable code points are sent to the remote side as '?'.
There will be an option for a single global default codepage. This will be used to render the C0 and 8-bit region of DEBUG and TTY and the translate tables glyphs when q_status.codepage is DEC. (Further detail: qodem will never support Unicode for emulations that do not have a clean method of handling it. The Unicode standard says that it is in essence a wire protocol so could be used anywhere. However Avatar can potentially use 8-bit bytes as part of the protocol by way of sending raw keyboard scan codes, ANSI.SYS has waaaay too much BBS-era CP437 use to permit it to do UTF-8 too, and VT52 formally specifies itself as 7-bit. Until something like a UTF-8-aware Wyse60 emulation is added, practically speaking the only Unicode emulations are also DEC emulations.)
The translate tables only apply to terminal mode, scripts, and host mode, EXCEPT for ASCII file transfers. So one can use the 8-bit table to break UTF-8 encoding and decoding, emulation parsing, and Zmodem/Kermit autostart.
ASCII transfers will use the codepage of the current connection, whether that is 8-bit or UTF-8, both when receiving and sending files. Another special case: if ASCII transfers are using an 8-bit codepage, then they will read the file as binary and send it directly on with no UTF-8 conversion. In other words ASCII transfers CANNOT destructively convert maybe-valid Unicode into '?'.
Host mode will have an option to use either the global default codepage or UTF-8. Messages will be stored in UTF-8 regardless.
The phonebook entry fields will display in the specific entry codepage. Keyboard macros will display in the current connection codepage. This will mean codepage awareness in fieldsets.
Wow, there are a lot of details here. But the net result will be really nice in the end. This will deliver features like the following:
Support for new 8-bit codepages (e.g. EBCDIC) can be added by the user with a new 8-bit translate table. qodem will ship with an EBCDIC table to provide an example.
Support for "pseudo-canonicalizing" Unicode code points, for example converting all of the arrow forms to a single arrow, smushing all of the smiley face emoticons into a single smiley face, or converting all of the things that look like an umlauted vowel into an ISO-8859-1 code point. This will also be used internally for the keyboard-to-8-bit-codepage conversion, by first converting to the same code points in the existing codepages tables and then emitting the index of that code point to the wire.
The ability to convert an 8-bit codepage to Unicode, for example turning Wingdings into emoticons.
The other nice thing is that by defining the data pipeline and noting where the conversions occur, it should hopefully be straightforward to get qodem to do "the right thing". If you want to operate on bytes, use the 8-bit table; if you want to make glyphs look right on screen, use the Unicode table. There is some overlap here: the Unicode tables could be used to do the same glyph-to-byte conversion that the byte tables do, but along the way would be pushing code points that are SUPPOSED to be ISO-8859-1 and pretending they are some other codepage -- the UI I will provide would make that ugly to work with indeed.
OK, let's make this a roadmap and get that into git. It will be maybe a couple weeks at least before I can really crank on it, but I won't call 1.0beta done until this is in.
August 8, 2015
I've been slammed with a major project at work the last four weeks, leaving little time to dedicate to qodem. However it is not forgotten or fully idle, even now.
A user submitted some bug reports on github, which is very nice to see. I've addressed one of them already (up/down wrapping around in the codepage pick dialog in phonebook), but the other significant one is a much bigger issue, namely how should we handle keyboard input to non-Unicode hosts? This really drives deep into the "Output" part (with respect to the user) of the whole I/O paradigm, and will impact at the least: capture file, keyboard macros screen, local echo, keystrokes themselves, and translate tables. So let's get going on defining the questions that need good answers:
In what order should the following operations happen to a byte received from recv(): process through translate table, written to capture file (raw), written to capture file (normal/html), converted to Unicode, modem parity handling, displayed on screen?
In what order should the following operations happen to a Unicode keystroke received from wgetch(): process through translate table, local echo, local echo (DEBUG), modem parity handling, convert to UTF-8?
Same as #2, but regarding a wchar from compose_key() ? Should compose key bypass translate table mapping? Also, Compose Key is more commonly known as Alt Code, it should be renamed anyway.
Should keyboard macros be aware of codepage? If yes, how? Each filename has both emulation and codepage? Or: macros are encoded as Unicode always but converted in post_keystroke() to current remote codepage? Should the keyboard macro editing screen show 8-bit glyphs (ISO-8859-1) as the current codepage instead?
Parity processing can already hose file transfers and Zmodem/Kermit autostart, by definition. Should translate table be able to do the same? If yes, then it can also break UTF-8 encoding.
Should translate table behave differently for Unicode hosts? Should it be capable of breaking UTF-8 encoding?
What should happen when a Unicode code point that can match the glyph for an 8-bit codepage comes in from the keyboard? Should it be mapped to the 8-bit glyph equivalent (e.g. perform the reverse function of codepage_map_char())? If so, should that 8-bit char have multiple Unicode code points backmap to it (e.g. the gazillion forms of left arrow become 0x1B)?
What should happen when a code point comes from the keyboard that has more bits than any available mappable 8-bit char? Mask the higher bits off (and then do what?)? Ignore the character? Warn the user?
Should the username and password fields as rendered by the phonebook UI be codepage aware?
Should the translate table UI screen be codepage aware?
I think this is a reasonable start to the questions. In answering these, what I want is for users to have sane defaults for both the classic BBS users and the modern Xterm users.
I could do the same underlying assumption as Qmodem(tm) did: the codepage is a system default setting, make that codepage really easy to use and the rest of the codepages possible even if not as easy. This could be done via a default codepage option (which would be CP437) that controls the UI screens, and then backmap keystrokes into remote codepage in the terminal. The keyboard macros and translate table screens would use the default codepage, and the username/password in phonebook would use the phonebook entry codepage.
The other extreme is to make codepage really visible everywhere. Maybe have a current UI codepage that is switchable any time in all codepage-aware screens. That would be cool to put in something like the TradeWars CIM code in a macro in CP437 (using compose key) and then seeing how that looks in other codepages including ISO-8859-1.
Let's let this noodle for a bit. I'm already seeing that I need to rename compose_key() and create two different codepage_backmap_char() functions (one for rendering a wchar_t to UI and one for sending a unsigned char to the remote side).
June 12, 2015
I've got cryptlib compiling under BC502, though I had to put in some fugly workarounds here and there. Since my qodem is currently using the CppCompiler I also needed to extern "C" in all my headers so that cryptlib can call back into qodem. The code is still not quite working, but I'll go ahead and commit at this point.
June 11, 2015
Fixed a lot of one-off things between Avatar, Avatar SGR, and Avatar ANSI fallback modes. Also moved the hardcode references to the scripts directory and the stderr fifo out to the options file. Along the way I think I just fixed the scripts quicklearn bug, we'll see if it comes back.
June 3, 2015
Took a week off for various personal things. Now it's time to look at bringing back cryptlib.
OK, I have a basic SSH client working again. cryptlib still needs some patches to handle environment variables and terminal resize, but I think I can see how to do that. It will be somewhat ugly, breaking a lot of cryptlib's internal API, but good enough for me.
I've updated the licensing language to credit cryptlib, and the readme to note that qodem effectively becomes GPLv3 (Sleepycat license) when cryptlib is used but stays BSD/MIT when not. This will have the interesting effect that the Windows binary will actually be Sleepycat while the non-Windows builds will be BSD. Oh well, anyone is free to rip out the cryptlib calls in netclient if they wish: qodem itself remains public domain.
May 26, 2015
Linux is merged with vt100. Xterm has some screen artifacts, not sure why but I'll find it eventually.
AS OF TODAY, MAY 26, 2015, THE SOURCE TO QODEM IS DEDICATED TO THE PUBLIC DOMAIN.
May 25, 2015
vt100 code sweep is complete. Still need to merge linux.c into it, but probably won't get to that today. We've got squirrels in the attic to evict first. Tomorrow will be a vacation day and I will need some not-computer-time.
But overall we've made a HUGE push in the last four weeks. Most of qodem's technical debt is now paid off. We are nearing the point where I finally get to write all-new code again, oh that will feel nice.
May 24, 2015
There goes netclient. So the last thing is combining LINUX and XTERM into vt100, and then the code sweep is finished.
In other news, I figured out that serial port support on Windows is a lot easier than TAPI. It's getting rolled into this release, and Q_NO_SERIAL is getting removed. This means that Qodem 1.0 will be feature-complete against Qmodem 5.0.
May 23, 2015
We are ALMOST DONE with the code sweep. All that remains is zmodem, netclient, and combining up linux and vt100. Of course these bits are still about 20% of the total codebase, but who is counting? Oh right, I am: with the style change to 80 columns, the raw line count is past 91 kloc, even though the true code lines count (via ohcount) is 56 kloc and essentially the same as before the code sweep, which makes sense. On to zmodem now...
...whew! Zmodem is so huge! Kermit was quite big too. OK, let's breathe now, goodnight!
May 15, 2015
Code .... sweeep .... continues. Ugh, this is taking forever.
In other news, I am contemplating a rather large departure for qodem: licensing the 1.0beta as MIT or BSD license. Let's get honest: I don't have anywhere near as much time as I used to for this project (or ANY coding project). Knowing now the scope of it, I will likely never have a period in my life like the 2003-2006 time span in which I could do this again. Nor will I have another job involving RS-232 where qodem becomes absolutely indispensable. Someday I will need to be completely off of it, and I feel like GPLv2+ would be just another barrier to someone out there wanting to poke at qodem in the future. Given how few people are interested in this kind of stuff, more barriers just means Qmodem's concepts die that much faster.
GPLv2+ was a decision made back when I was all gung-ho on "Free Software is totally different than Open Source" and "GPL is why Linux took off and the BSDs languished". I don't believe much of that anymore. Apache and nginx are examples of commercially valuable platforms that are BSD-ish yet still going strong in the community edition. clang/llvm are doing fine too. It wasn't GPL that made Linux work, it was the leadership and community and being at the right place at the right time.
It looks like my libraries remain OK with BSD licensing, so long as a) the changes I made to compile my libc bits for Borland are deemed public domain (I just added text to that effect in the CREDITS file), and b) the source to 1.0beta is made available. This looks good to me.
I'm not going to pull the trigger quite yet, I need to let it sit in my mind for a couple weeks. If I do it there's no going back: this would be giving the entire world -- including the sleaziest "entrepreneurs" I kept running into in Boston -- my largest single piece of work. But I also know that programming is not my long-term professional future anymore. Things are different, and I think it's time to take this last apron string off this baby.
May 14, 2015
Code sweep continues. 23 files to go...ugh. But on the bright side it is causing my eyes to bleed a lot less when browsing the code.
May 12, 2015
Code sweep continues. Debug emulation is now refactored into emulation.c since it is so small. This has been bothering me for some time, I guess it was seeing that debug.c that had nothing to do with debugging. Which prompted me to also decide on eliminating all of the DEBUG_X #ifdefs in favor a simpler dlogprintf() type macro. This will reduce line count, de-clutter a lot of source, adds a timestamp, and unifies all the different sources into one output log.
May 10, 2015
The headers code sweep is finished. I found several loose ends for later, they are now in the TODO. I'll commit this now, but I'm sure I broke the Borland build in qodem_write(). We'll check that later on.
May 8, 2015
On the code sweep, we are almost halfway through the headers. I've been cleaning up as I go along. I took out some never-called functions in field.c, made dialer's functions return void since no one cares, moved dialer_get_term/lang over to emulation, and finally nixed status.h is gone (it's had that for a long time coming). Noticed a different kind of failed-connection bug so fixed that -- basically I was assuming that if DNS lookup succeeded then socket() and connect() would succeed, but connect() will die immediately if the network isn't available. Checked both Linux and Windows builds, interestingly enough Linux returns EINPROGRESS for the same condition Winsock returns EWOULDBLOCK.
This sweep will take longer than I thought. I've also got some real life catching up to me again, and tomorrow need to spend a not-computer-day. I'll just have to keep chipping away at it over the next couple weeks.
May 7, 2015
I've got a GNU indent profile that is pretty close to what I want, so I am starting the code sweep now. This will take some time, but hopefully will be done by the end of the weekend. After that I think comes a regular Borland-based build and an announcement.
It would be fantastic if I could find someone out there with an actual modem who could help me get host-mode modem support working.
May 6, 2015
The libz/libgpg_error/libgcrypt/libssh2 build path on Windows really sucks. I'm trying CryptLib again instead. On the downside, this will make things take longer to 1.0beta, but on the upside we get the SSH host server back. I remember now why I moved away from it: tty terminal resizes couldn't be passed through its session interface. Others have encountered the same, but they have also patched cryptlib's ssh2 to pass more reasonable values. I can put in some custom peeking from cryptlib into qodem values if necessary to get what I need.
May 5, 2015
More bug fixes. Crashes eliminated in uploads, keyboard macros, hangup, and host mode. In general it's looking better, but it does highlight another class of bug: calls to POSIX API's that can return EBADF's for the most part will crash on BC5.
May 3, 2015
More bug fixes, this time in net_connect_finish() calling dial_success() without setting pending to false, resulting in the program state being stuck in Q_STATE_DIALER and thus missing the beginning of input (in this case the login banner) until the user pressed a key. Sometimes. (I've thought about refactoring all the q_dial_method and telnet_read()/telnet_write()/etc calls into an OO struct with function pointers, the way putty does it, but until I get another several ways to connect it doesn't buy me anything new except another round of testing.)
The Windows display also looks a lot better now thanks to switching to Classic Console font and gamma-correcting the CGA RGB values. This font loses out on DEC special graphics characters though. I'm very tempted to learn enough font stuff to add those glyphs from Glass_TTY_VT220 to round it out.
May 1, 2015
Lots of semi-minor fixes today. The options are now sorted in a much more logical fashion. Also my colors are finally correct (I hope!), turns out I was mapping all these nice bold-as-bright colors but not actually using them, so the "brown" that I saw was entirely an xterm thing and not a qodem thing. This also makes the screen API quite a bit simpler. I wasted a couple hours not remembering -- and having to dig into the PDCurses source to see it again -- that init_pair takes a number between 0 and 1000. Oh well. A long day and I think the codebase is actually smaller. Woo.
April 30, 2015
I found a number of small glitchy things in the Borland build to fix. I'm surprised that host mode worked in Linux at all actually, as I was calling freeaddrinfo() right BEFORE referencing it. Yay for Winsock giving me an error I could trace. I also had stupid logic in set_nonblock() that was made the net listen socket blocking, fixed that and now I can telnet to a qodem host.
Next up is figuring out why I sometimes get completely stuck in Windows. It seems like switching out of foreground and back is doing it. My guess is that I am missing a message of some kind in qodem_wgetch(). (It's nice having breakpoints in my editor again, I do like Borland.) After that comes building the stack for SSH support, and then a huge slew of other things.
April 28, 2015
Well! Three years eh? Life has had its way with me. About six months after the last posting we started planning a move to another state with a new job and career track. Roughly ten months after that was interviewing to return to my normal (non-programming) career track and another cross-country move. Six months later was moving yet again into a house. This is actually one of the longer times I've been at a stable address, so I finally have some mental space available to get back into the groove.
In the last three years I have done a few interesting side projects that are coming back around to benefit qodem. I ported kermit to D, and found some actual bugs in my byte encoding which I ported back to the C code. I also built a windowing system like Turbo Vision, first in D and then done better in Java: it's over at https://github.com/klamonte/jexer . As part of that project I have a very clean VT100/VT220/XTERM emulator, basically the grandchild of qterminal, and it's quite nice actually. I've also been picking off small bugs off-and-on, driven by a very long-running TradeWars 2002 game. (Let me also say that having over a HUNDRED TRILLION credits in TradeWars kind of ruins the game: I don't think a helper can even spend it faster than the interest builds, even with buying 100,000 figs at a time and dumping them ASAP.) I also had a job of doing packaging for both deb and rpm, and have learned a lot more about autoconf than I ever wanted to know before.
Perhaps most important though is the use of git and github, these are really a game-changer for open source projects.
So a couple weeks ago I got saddled with Office 2013 including Visio 2013 at work. Long story short: they all suck ass. Visio in particular had so many new bugs that it killed my engineering workflows. It was bad enough that I decided to put together a Win2k VM with Office XP and Visio 2002, just so that I could have some kind of get-out-of-jail key to get work done. Plus Win2k is soooo much faster to load than anything since, it's ridiculous; even my little Acer Aspire One netbook can run Win2k almost as fast as my high-end desktops back in my IBM days.
I noticed that the Win2k VM I had setup a few years ago had Borland C++ 5.02 already on it. I really miss Borland C++, I had picked up this copy in college and had some very fond memories of writing a DirectX game with it. It also happens to be capable of making executables that will run on Windows 98. Wouldn't it be really nice if qodem could run on Windows 98, maybe even Windows 95, and every version of Windows since?
In the last three days I've got qodem compiling on Borland 5.02. And today I was able to telnet to a localhost Synchronet node in glorious ANSI color. The new code repository is on github: https://github.com/klamonte/qodem .
I'm doing it: we're getting the 1.0beta release this summer. Windows is becoming a first-class qodem build, with a fully-Windows-hosted toolchain. We'll start with Borland, and possibly a Visual C++ build too if I can obtain a copy of 6.0. The autoconf build is getting re-written to be much simpler, and both the X11 PDCurses and mingw32 builds are going away. Our primary targets are ncurses, and the Win32a PDCurses build.
The code is on github and I would be thrilled to get pull requests.
May 30, 2012
I've been fixing enough bugs the last week that I went ahead and version-bumped CVS to 1.0beta. It will be another month at least before I put this version out on Sourceforge, as I'd like to wait and see if any other bug reports come in. I'll probably do the same as last time though: put a development download link for a while, then maybe 3-6 months later roll it out as a full release.
A big part of the next release will be generating finalized documentation and more thorough testing of each feature. This is a long slog but it's been about 6 years since I last did it. I know things like Zmodem downloads work well because I use them all the time, but what about things like "compose key + split screen" or "find in scrollback": I hardly use them so there could be some interesting gremlins lurking.
This week I've been focusing on crash/segfault bugs, of which there were a surprising number in the phonebook dialer which got exposed when I routed the network connections through it. OTOH, fixing those also makes the modem dialer more stable which is good too. I've also finally found and fixed some FreeBSD bugs from a while back, one of which would have gotten other users ("\e[;H" as the very first ANSI code) and another was just a brain fart in one branch of Xwcsdup() that might have gotten other non-glibc platforms.
I finally sifted through my old BBS logins. Some boards I hadn't connected to in almost two years. About half are gone now, either the board itself vanished or my username did. Time soon to bring in another board or two for a nice TradeWars game.
May 19, 2012
Today is release day!
Last night I switched to the win32a build and it looked very nice. given that the only feature still out there is host mode modem support for the original development plan, and everything else seems to be working, I'm ready to cut here and get this out there. The version is 1.0alpha, with a total code size right near 80,000 LOC, or 37% larger than 0.3.2. This has by far been the most significant code push of qodem's entire history.
Noticed that Zmodem downloads were corrupting files thanks to Windows' default behavior of corrupting line endings, fixed the fopen() calls to pass 'b' in both Zmodem and Kermit and that looks good.
The only thing I have now is to upload the files to Sourceforge and post the update on Freshmeat. But lunch first...
May 18, 2012
In an interesting twist, network connections now go through the "modem" dialer. This allows redialing through network systems, and also is a lot prettier. Getting the code there exposed some other glitches that are now fixed.
Added the modem host strings and put modem lights (RI, CTS, DTR, DCD) on the alternate status line. Changed the hangup sequence to drop DTR first before sending the hangup string. I can't test it though, I need a phone line. Beyond testing, the only thing left on modem support is host mode picking up the phone. Not bad. In fact, that is that LAST actual feature between this place and 1.0 alpha.
Finally, I tracked down the scripting bugs: besides some conflicts between script_start() and script_quicklearn(), there was a race between script stdout and stderr: if stderr closed first, I was never going beyond in script_process_data() to read from stdout and eventually call script_stop().
May 17, 2012
The known_hosts logic is working. It took some extra work to pull the OLD MD5 hash of the server key in known_hosts so I could ask the user if they wanted to overwrite it. I had to add a base64 decoder (public domain libb64.sf.net) and include libgcrypto for MD5. It works out since I'm using the same crypto as libssh2. It also didn't help that libssh2's examples did not cover actually adding a key to known_hosts, I had to read their source to see that I could pass NULLs into the comment and salt fields for libssh2_knownhost_addc(). But it's working now.
I also just added the password login to host mode, which is really nice. While testing the protocols I noticed that Zmodem was the only one actually working, which is nice because it will guide me to getting X/Ymodem and Kermit going now.
.....aaaand with this commit I broke the 80,000 LOC milestone: 80159 lines.
Time for bed. I finally got the protocols working through host mode. It was a bit strange but makes sense now: when host mode switched to file transfer modes, the checks in qodem.c to call telnet_read() / telnet_write() were failing because we weren't "dialed out". Adding the q_host_active flag made it call telnet_X() and socket_X() as appropriate. And now we have Xmodem and Ymodem going through, in addition to the Zmodem and Kermit going through earlier. There was also a sequence bug in Ymodem's 0 block response to NAK to fix, and Kermit's NAK to the initial Send-Init. With these changes plus the password, host mode is shaped up very nicely indeed.
May 16, 2012
A very old bug in Zmodem is now fixed (and it was absurdly easy): after 15 consecutive bad header blocks, the transfer aborts. I also put in a notification of a TCP connection because otherwise it looks like a hang when it's just waiting to get the socket.
Kermit RESEND support appears to be working now. The protocol documentation at ftp://www.columbia.edu/kermit helped a great deal.
I have about 80% of the known_hosts code in too. I just need to base64-decode the host key so I can show it to the user, write new keys to the file, and pick up the option from qodemrc to point to ~/.ssh/known_hosts (on Win32 that will be in Documents\qodem\known_hosts).
May 11, 2012
Break time! SSH connections are working through Win32 now. I switched to libssh2 and gcrypto in the end. I think what was actually killing it was the -mwindows flag that libcrypt/libssl needed; qodem is a console application but libcrypt was referencing CreateDC and other GUI functions. But that's still a guess.
For the first time, I'm actually putting a preliminary Win32 build on the download page.
Code size: 79412 lines.
May 9, 2012
ssh connections are working through libssh. The API is a little different from what I expected. A nice clean exit will appear in our normal ssh_channel_read() returning 0 -- which follows the standard POSIX convention -- but if the low-level is killed before the nice SSH handshake is completed, it returns -1 with an SSH_FATAL error. So I had to catch that too, internally returning it as EIO like a write to a closed socket. The main problem was that if I didn't catch this correctly the select() would return EBADF. But it's cleaned up and appears to be working now.
I managed to get libssh 0.5.2 to "compile" under the Win32 cross-compiler, but the resulting qodem executable does not run (immediately terminates with no output). This is a bit aggravating. The libssh seems to work OK on Linux, and the code is relatively clean with only a few modifications to get it to compile. But the resulting binary doesn't actually work. Grrr....
May 6, 2012
I discovered the joys of Haiku OS in a VirtualBox VM. It's pretty cool actually, I'm very tempted to port PDCurses to Be/Haiku and producing a native binary for qodem. Conveniently, the Alt- keystrokes are already used by the Terminal application, so this shouldn't be too different. Since Haiku is still (sometimes) using the gcc 2.95 compiler, I had a little bit of work moving data declarations back up to the top of functions and blocks. But gcc 2.95 was actually qodem's first compiler, so it wasn't too hard to get it fixed. Not linking fully though, the ncurses in haikuports isn't the wide-char version.
The nice side-effect of this work is that I updated Makefile.old to be a vanilla ncurses-based compile. I've forgotten how much I miss the old "just-compile-it-already" days before autoconf and automake. It's been nine effin' years since I could just "make" without "configure".
May 2, 2012
I've been picking things back up again. The last few days I put in compile guards around all the serial port code to isolate it from the Win32 port code. I'm planning on going to 1.0 without TAPI. If Win32 users come in later asking for it I can slot it in later, but most people out there are asking about BBSes and ANSI in relation to telnet and ssh systems.
I've also started migrating from CryptLib to libssh. There is still more work to do, but most of it is the initial network connection, the other polling and port management is the same.
In the meantime my tools have changed a bit. gcc 4.6.3 introduced a new warning (-Waddress) that flags on attr_get(), so I can't compile with -Wall -Werror anymore. PDCurses also hardcodes X11 directories that no longer work, so it needs them passed into configure. It also segfaults during Xinitscr and I don't know why. Renaming ttytype to pdc_ttytype seemed to fix it, but that shouldn't have worked. I probably have a memory bug somewhere to hunt down.
So where are we now? Things are a bit broken in some spots, but we're getting closer to a working 1.0 release candidate.
October 28, 2011
Windows support continues. LOCAL connections are working, but cmd.exe is pretty crappy about leaving the "console" (pipe) in cooked mode, so you have to press enter before it responds. Scripts are running too, and exposed a timing bug with the POSIX scripting. It's on the list, I'll have to get to it later.
The only major Win32-only codepath left is serial port support. The rest is debugging and plugging in SDL, UPnP, and CryptLib.
I'm going to have to back off a bit due to increasing work in my real life. But for the first time in qodem's history, I'm down to a long list of things that can be tackled in small increments. No more deep dives into mountains of function. One bug at a time, and maybe in 2012 I can wrap this up.
Code size: 79257 lines.
October 22, 2011
It is a big day for qodem today: for the first time ever, qodem connected to a remote system, survived vttest, downloaded a file via zmodem, and did all this while running under Windows XP.
October 18, 2011
Figured out the stalls, it was a few small things. packet_buffer wasn't being looked at on the entrance to every loop; decode_input_bytes() needed to flag if it had successfully taken a packet off of packet_buffer so that we loop again; and find_output_slot() was giving the wrong slot during a check_for_repeat(), leading to potentially always stalled behavior on more than one sent file per upload. Those are now fixed and a new alpha released.
October 17, 2011
Kermit full duplex sliding windows are in! There is a stalling issue with sending, but transfers still make it. And both sending and receiving can survive some pretty atrocious channel conditions. And I found the debugging problem: some logic for extended packets was stuck inside a debug guard.
I'm going to update the dev snapshot on the web page and park it for now.
Code size: 77736 lines.
October 16, 2011
Kermit is getting pretty close to done. It still has lots of testing, but sliding windows are basically in and working as they should, at least on reliable links. There is at least one corner case somewhere screwing up transfers -- it vanishes when I enable debugging which is a pain but I'll get it figured out eventually. I also got a quick 7-bit link check in for the parity behavior: it will always seek to NOT do 8-bit prefixing unless it sees a serial link using 7 bits, e.g. 7E1. What remains is beating it to death on unreliable links and verifying that out-of-order packets are handled correctly, and then adding RESEND/PSEND capability, and finally MAYBE putting in locking 8 bit. I know one simple heuristic for locking 8 that wouldn't be too bad:
1. Load up a block and encode it. 2. Count the number of 8-bit prefixing done in step #1. 3. If more than 55% of the bytes are prefixed, rewind the file, flip the toggle, and then re-encode it again.
But beyond pedantry I'm not sure how useful it would be to add it at this point. I barely even have a serial port setup anymore to test 7-bit links, and all of my future plans with the Kermit protocol are on 8-bit links with both high latency and high throughput (hence the desire to get windowing working OK).
All this work now makes kermit.c the second-largest source file in the whole project. It even beat out vt100.c, which is saying something. But on the whole, I have to say I am much happier implementing Kermit than any of the other protocols. Granted, I know now how to make zmodem.c re-factor a little better, but honestly Zmodem as a protocol is just hard to make very robust because it bleeds state between so many layers. Kermit has some trickiness with the SEQuence number, but overall it is well-divided between marshalling/encode/decode and packet state machine.
October 2, 2011
Ah, the ever-so-elusive 1.0!
A lot has happened over the last 11 months. Qodem has gained many features, including a help system, host mode, internal implementations of the protocols (telnet, ssh, rlogin, raw socket), and the beginnings of a port to Win32 based on PDCurses. It's also had a major overhaul of the web site using a theme based on the Elinks user manual. And several kind folks out there have helped get it into the Fedora repositories, identified bugs, and helped with the Qmodem(tm) archive.
On a personal note, I've also changed jobs and moved from Texas to Michigan, which put a 6-month hiatus on development, and also took away my POTS line so testing modem features will be a bit complicated.
So where are we now?
Last year I was diligently doing my normal thing of working on one new major feature between releases, when I went to bed one night and the next morning realized that I knew how to sketch out all of the remaining big features leading to the 1.0 release. These were the help system, host most, internal protocols, and the Win32 port. The big breakthrough was getting the PDCurses build working. Once that was complete there was a clear path to the Win32 build. Host mode was just another finite state machine; the help system was just another full-screen dialog; and the Win32 build was made much easier with the mingw cross compiler script. So I decided that even if would be a really long cycle, I would just make the next release the final 1.0 release.
I'm now down to basically just finishing up Kermit (always with wrapping up Kermit!) and finishing the Win32 port. The Kermit code waits mainly for me to get a really good window at finishing it up for good, because I intend to re-use that code in several other future projects and I want it as high quality and ready to transliterate to Java and D as I can get it. The Win32 port is already underway, with the main hurdles being the file and network I/O. These are somewhat a pain, but can still be worked on one feature at a time.
It's still many months away to that 1.0, but I think it will be worth it. If nothing else, I'm ready to get my hobby time back. :) I'm generally enjoyed working on this project, but being a perfectionist I keep adding one more thing and moving my own goalposts.
October 31, 2010
I see the way now.
October 29, 2010
I've got two more days I can push hard on qodem before I have to back off and take care of real life again. I've decided to focus on these items:
* Making Kermit work over unreliable links. * Fixing the last known Zmodem issue (a delay in seeing SUCCESS at the end) * Re-testing the dialer and modem hangup on a real dialup system. * Printer support
Once these are in, there are only three more releases between 0.3.3 and 1.0 on the roadmap. Realistically, that may only be 1-2 years away. And 2.0 (the port to Windows) may only be 80-160 hours beyond that now that the PDCurses build is working. This is pretty amazing seeing the light at the end of the tunnel on this project.
[ ... later ... ]
Today has been really productive:
* I've got the phonebook 80/132 print function working, well sorta: it creates a text file and spools it to lpr if the output is "|lpr", otherwise it generates a "savedfon.txt" file in ~/qodem for the user to work with later. * The number pad now works, including keyboard macros, thanks to PDCurses. * I have re-mapped the colors by default to match the DOS colors. This works both with the PDCurses X11 build and the normal build running under both XTerm and Konsole.
Tomorrow I'm hoping to fixup Kermit and also actually try to dial a real modem and see how the dialer has held up. (It won't be too long before an analog line capable of supporting a modem becomes a specialty item.)
But it's time for bed now.
October 27, 2010
The last couple days I've been working on some features I've been missing that seemed too hard to get to earlier. The first was customizable colors. I was expecting a real mess of qodemrc options, but then I thought about how multimail does it and figured out that it would be trivially easy to do the same. And now there is a colors.cfg file in ~/.qodem that has templates for all of the original Qmodem(tm) color themes. It's pretty bad-ass to switch to the "Red Shades" after all this time. :)
The other feature is a little more dear to me recently: I have an X11 build! Woohoo!
This means that I no longer have to fudge with Unicode fonts, default foreground and background colors, metaSendsEscape, and 80x25 hacks. In short, everything in the Getting Started Guide is obsoleted by the X11 build. I spent a couple hours today figuring out how to make a separate Debian package also so that one can install both 'qodem' and 'qodem-x11' side by side on the same system. Both packages so far on Squeeze are lintian clean. It will be a little more work to maintain two man pages, but otherwise everything is shared. And it also caused me to make fixes to my semi-broken ncurses-only stuff, so things look even better in the source.
I'm going to be tempted to roll out 0.3.3 quickly and push the big regression to 0.3.4, but I'll have to stand firm. It's been probably four years since I've just been a plain old tester of this project and I've got to exercise it more and get a really stable build out there next time.
Overall though I'm pretty happy with where things have gone this week.
October 25, 2010
This weekend I released 0.3.2. It's a pretty good release. It fixes a crash bug (from a double-free()) in the phonebook 'F'ind, exposes the port for ssh and telnet, and fixes a bug in keyboard handler that prevented communicating with the modem when not online (pretty big one for serial port uses). I made really good use of Jeff Gustafson's rpm spec file and built a Fedora FC10 build (which is quite popular on the download page). (I noticed after the build that Jeff's credit was only in the CREDITS file and not the README - I fixed that in CVS.) Right now I am downloading the amd64 version of Ubuntu 10.04 LTS Desktop to boot up a live CD and compile for Ubuntu 64-bit. After that I'll see about spinning a OpenSuSE 32-bit rpm and eventually a 64-bit Fedora rpm.
That's a lot of different builds! The reason for so many builds is that I'm getting really close (in total project lifetime terms I mean
- it's still a few years away) from having a really long-term stable build. In fact 0.4.0 might just be that build. It will have essentially all of the features necessary for a good modern terminal emulator. After a very serious regression test effort (coming in 0.3.3) I may be able to mark this project as "mature/stable" and move my primary efforts to my next project (codename NIB). So in about a year I expect to take a bit of a breather.
It's been a really fun project overall though. I've gotten very close to this technology, and still see some potential for bringing some other BBS concepts back to the Web 2.0 Age. I've become a better programmer, learned some humility along the way, and might have made some other people happy too. It's been very good for me.
September 16, 2010
I just rolled another release to provide the baseline for (crossing fingers) what may become the first version of qodem in the official Debian repository. Martin Godisch, one of the maintainers of minicom and lrzsz, has been helping me get the packaging up to snuff and very soon I'm hoping we'll have a sponsored package uploaded.
This release also AFAIK fixes all of the remaining issues in the Zmodem implementation. Block sizes go up and down based on two different heuristics (8k of good data: UP, 3 bad blocks: DOWN to 32 bytes/block, 10 bad blocks at 32 bytes/block: give up). I found the unreliable link bug: my crc flag was getting reset to CRC16 by the receivers ZRPOS and not being pushed back to CRC32 on the next ZDATA. Totally not what I expected the bug to be, but fixed and now files get through even with significant "line noise" (faked with random data). So Zmodem is looking like my best protocol. The next major release I'll get Kermit features implemented to beat out Zmodem :) .
I also finally fixed the View Directory window so F4 will hide/show dot-files, and regular alphanumerics will select the next file/directory that matches that letter. Other fixes included sgr() blink and underline in vt100.c (linux.c had it right - I must have been tired on the RV Gyre when I wrote that code), SPACE parity (which is still not great but I'll get there in 0.4.0), and rendering of the console status line.
Overall I'm getting a nice feeling about Qodem again. It was pretty dicey going through the UTF-8 conversion - lots of things were exposed as broken: codepage handling, input, and the screen. But I've had very few hassles with Unicode now and I'm back to putting in new functions and rounding out the old ones. It almost reminds me of 0.1.1 which was a really decent long term release.
Code size: 58574 lines.
September 10, 2010
Nothing like a full code sweep changing bool/true/false to Q_BOOL/Q_TRUE/Q_FALSE to remind me of how big this thing is nowadays...
After several years, I have finally noticed the way to always get good behavior out of xterm. Now I have white-on-black working AND get metaSendsEscape by default. Once I verify that qodem can survive 80x24, I'll be down to only one issue (Unicode font) that would hit people with a stock xterm install. Of course, the bad UI issues users saw was completely my own damn fault: xterm and ncurses both had everything I needed, I just didn't research them thoroughly enough.
August 14, 2010
I am in the final stretch to a major release. It works (well, compiles anyway) on FreeBSD now, and OS X, and armel Ubuntu (Sheevaplug). The many small details (man page, versions, copyright statements, disable debugging, etc etc) are done.
I put up a Pootle translation server at qodem.dyndns.org:8080 , and I re-extracted all the English strings because so many of them have changed with the scripting support.
I'm going to build the Debian i386 and amd64 packages and then let it sit for a couple hours to see if I end up remembering anything else. But...it's feeling good. I can't wait to get this out there and then start writing a TradeWars helper. :)
Code size: 57807 lines.
August 8, 2010
This week I finally figured out the LAST major feature qodem has been missing: scripts. The design is so simple I can't believe I waited this long to think about it... I'm about 30% of the way there.
Basically, scripts are just processes spawned just like the connection programs in dial_out() via forkpty(), and qodem sits in the middle and passes post-processed data back and forth between them. The script's stderr is routed to a FIFO and shown in the bottom of the console window to report messages to the user.
This design has several awesome advantages:
Scripts can be in any language.
Scripts just have to read from stdin and write to stdout for their API.
A complex script can be developed and tested independently of qodem, and can use the most advanced techniques available. Imagine a TradeWars helper communication through qodem but having its own I/O through X11.
Qodem handles all of the emulation details, stripping escape codes and control characters before passing the characters to the script's stdin.
With very slight changes (mainly simplification) the scripting infractructure is exactly what we need for external protocols. Someone could easily add sexyz, gkermit, or (if I ever get to it) HS/Link.
About the only thing we lose in this design is the ability to script qodem itself: dialing other entries, changing emulation or toggles, etc. That can be rectified with a more complex handshake to the script (maybe something like telnet IAC codes), but there might never be user demand for it so I can wait.
I've spent years dreading scripts, with a spec and the desire to learn lex and yacc and make a minimal-but-decent "VM" to run scripts in. And then I realize that I'll never need that in my real work: just a dump pipe that strips the emulation out and passes to another process is plenty for me.
I'm both tired from coding - it'll be another week before it's testable - and relieved that it's finally in sight. After this feature is in and working I'm going to cut a new release and pause for a good long while...
July 19, 2010
I went ahead and wrapped the 0.2.1 release with a tarball and amd64 deb. I'll get the RFS email out and see if maybe this time someone picks it up...
July 10, 2010
Whew, lots of small updates:
I added explicit support for multiple codepages, so ANSI BBSes might be CP437 or ISO-8859-1, or in the future ATASCII/PETSCII/etc. The UTF-8 and DEC terminals explicitly set their glyphs so they do not have their own codepage settings.
Many of the runtime toggles are now settable for each phonebook entry. That's handy for when one BBS needs line wrap but another doesn't.
ANSI.SYS now recognizes and discards the RIPScript detection sequence (CSI !).
A new method for explicit command lines was exposed to facilitate connecting through proxies. You can say connect with the CMDLINE 'tsocks ssh firstname.lastname@example.org' for example.
Some of my OSX modifications are now included. It seems to compile and run mostly OK on my Leopard box.
I'm getting very close to calling a 0.2.1 release. I think I'll keep testing a couple more weeks, but the next RC is going out on the web site now.
Code size: 56060 lines.
June 14, 2010
I'm running under amd64 text-mode only. The zmodem crc32 code naturally failed; changed unsigned long to uint32_t and it immediately started working again. Version bump to 0.2.1 while I test other issues under amd64.
December 18, 2009
I just uploaded 0.2.0 to debian-mentors. Hopefully it can find a sponsor soon and get any Debian issues worked out. I've also contacted Jeff Gustafson regarding some packaging work he did for Fedora. With any luck the "final" 0.2.0 will build out-of-the-box for both Debian and Fedora and behave well.
I'm really excited about this release. (Of course, 10 seconds after uploading I'm adding a minor bug to the TODO, but that's how it goes... :) )
December 17, 2009
It's been a mad dash to get all the function I wanted in place for the next release, but it's here now! Unicode pretty much everywhere except filenames, a slightly cooler info screen, fixed form handling, and a few glitchy things better. I'm now switching modes to documentation, working on the man page. Once that's in I'll be in test mode for at least a few weeks. These last two releases have touched pretty much everything and need to be more thoroughly looked at.
I changed the numbering scheme such that this next release is 0.2.0 to designate point releases as fixes for downstream.
Code size: 54605 lines. (I'm not even sure why I'm keeping count anymore. Habit I guess.)
December 12, 2009
Whew! I finally got a decent (I hope!) Debianized build going. Getting the "ITP" (Intent To Package) email out took some effort too: reportbug (3.x) in Debian stable barfed due to UTF-8 issues, so I wanted to use debian-bug in Emacs, but that required getting my local email to run over SSL (solution: stunnel4 + exim4 changes). But now it's working and my development system can now send email from home. So I got the ITP bug sent in, but I don't know if that's enough to get the process rolling or not.
Getting it ready for Debian involved finally updating the man page to something more useful and cleaning up many other small issues with the build. No matter how much time it takes to get into an official repository, the project has benefitted very visibly from "Debianization".
I also separated the music sequences for connecting via modem and connecting via everything else. The nice thing about having the music was when I would be dialing for an hour or so and just wanted to be notified when I got in; the modern age has kind of made that obsolete. With this change I still get music for file uploads/downloads and "ANSI Music".
I'm awfully close to cutting 0.1.4 features-wise. I want to add Unicode to the phonebook and keyboard macros, fix field_render to behave a little better, add capture-on-connect to the phonebook, and put in a "mixed doorway mode" feature. I can probably knock two of those items out tomorrow.
I'm really getting excited about this next release though. It will feel very nearly like what I was trying to get from the start of this project.
December 9, 2009
I replaced the calls to libform with a very simple text field which more closely resembles the original Qmodem(tm) look. I'm also hoping it makes it easier for future support of wide characters and also porting to Windows.
Also, I got double-width and double-height lines working, and it was SOOO much easier than I thought! Go me!
December 4, 2009
I'm just about done with Kermit for the moment. It still lacks some more advanced features such as locking shift and long packets, but it's darn close in function now to gkermit and should do quite well over TCP/IP and reliable 8N1 serial links. Features for unreliable links -- mainly full-duplex windowing and better ACK/NAK behavior -- will be deferred to the next release.
In fact, that's all the 0.1.5 release will be: file transfer protocols over unreliable links. Zmodem needs to change its block size on errors; Kermit could benefit from having server mode when the other side is over a serial link.
I'm hesitant to wrap a 0.1.4 too quickly though. I've got a lot more shakedown to do with the Unicode and Kermit, and making sure the emulations are more solid than the 0.1.2_en release. I'm tempted to bring forward a Unicode-to-ASCII map so that you don't HAVE to run a UTF-8 console, but then I have a feeling that could be a step backwards. Maybe I should defer until I've had more time to eat my own dogfood.
For now I'll leave CVS head in a good place, test over the next few weeks/months, and then cut a 0.1.4.
Code size: 53237 lines.
November 24, 2009
I went ahead and cut 0.1.3 as an alpha release. The Unicode and Kermit support are pretty big and will need a couple releases to fully iron out, but it's so much better than the 0.1.2 release I couldn't hang onto it much longer. I'll probably cut a bug fix release next spring. I wish I had more time to dedicate right now...
Anyway, hope someone out there enjoys this.
Code size: 52122 lines.
November 23, 2009
Basic Kermit receive is mostly working. I have to admit, Kermit is a MUCH easier protocol to implement than Zmodem. The only "missing features" I've seen -- file modification time -- is a deliberate omission in gkermit, not a bug in the protocol itself. The documentation ("The Kermit Protocol" PDF floating around) is very good, and the state machine is very simple in comparison to the other three protocols. Plus when I get stuck the gkermit code is much more readable than lrzsz (but still looks like mainframe assembly language).
I'm basically only one feature away (extended packet format) from having a decently fast receive implementation. Then I'll get into the send side, and finally the more modern features (locking shifts and streaming).
November 20, 2009
Whew! I've gotten a lot done recently:
There is a new UTF-8 aware emulation for Linux. This allows Qodem to be a true Unicode terminal emulator.
VT220 NRC character sets and the Multi-national set are converted to Unicode. Plus a little tweaking and it appears to be doing the correct thing now in vttest.
I've begun stubbing in Kermit. I'm hoping to have a minimal Kermit implementation in pretty soon. From the docs it looks like it will be a hell of a lot easier to implement than Zmodem was.
I'm ready to get some sleep and look at Kermit tomorrow...
November 15, 2009
I've spent the last few days bringing over the QTerminal Unicode codepage support to qodem. Getting the wide-char ncurses output wasn't too bad conceptually, but it was a LOOONG slog through the 1000-ish direct calls to ncurses to wrap it with my own calls. OTOH this is a significant piece of porting to non-ncurses platforms, so that's nice. But what's REALLY nice is that all of the CP437 glyphs render right - even the music symbols I couldn't get in earlier - both on the Linux console and under xterm. (Took a while to find a good font that had both CP437 and the DEC special graphics characters, but Andale Mono did the trick.)
There's also been new activity on vttest exposing some broken behavior in VT100 and LINUX emulations. That's fixed.
I finally gave up on using gettext to translate everything by default. The en_US keys are now directly in the code. This is the normal case for most open-source projects, but now I have the ability to lose a string and make translation harder for anyone who might come along later. But at least it works whether or not my locale is set.
I've got a very long list of code sweeps, but I'm hoping within a month or two of making a 0.1.3 release.
February 23, 2008
I discovered an awesome project on SourceForge: Turbo Vision! The last couple weeks I've refactored the VT100/LINUX emulation code into a QTerminal window class that runs easily under a Turbo Vision application. It's quite slick: I can run Qodem-inside-QTerminal and it works nicely. Since this version has only a few Unixy things, I'm looking later this spring at porting it to Win32 too.
I'm hoping that the Turbo Vision maintainers might be interested in getting this into their project.
September 10, 2007
Found two bugs in Synchronet's Zmodem implementation: 1) it doesn't do ZCHALLENGE; 2) it doesn't do ZSKIP. They are both rather serious, I'll see if I can report them...
September 6, 2007
I found a bug in shorten_string() that caused a segfault, fixed that, added a check in save_form() to prompt for overwriting a file, and ...
...I just published the 0.1.2 release! Code size: 46683 lines.
September 5, 2007
Finally got around to checking out Xmodem and Ymodem vs. Qmodem(tm), and also finished support for the -G protocols. Xmodem with Qmodem(tm) works fine, Ymodem receive works OK, but Ymodem send (from Qmodem to Qmodem(tm)) is broken. Qmodem gives up immediately, and provides no clues as to why.
I'll try them out against ZOC and HyperTerminal, but won't spend much more time on them. They are buggy protocols and implemented in several other places (not just lrzsz) so other people have recourse if they need their own code. Zmodem OTOH seems to be much more rare, and despite its flaws it IS capable of sending files across without hosing their contents on 8-bit-clean channels so there is some real value in keeping it reliable.
But, assuming that I am nearly finished, I can just about wrap the 0.1.2 release next week. That would be very nice indeed as it fixes so many bugs from 0.1.1.
September 4, 2007
Found two bugs in HyperTerminal's Zmodem receive implementation. Both are now noted in the README.
Also found a SERIOUS bug in parse_packet() that would core qodem badly when Zmodem'ing with QmodemPro. And a bug in ZOC (it does not understand ZCRC) and one in Qmodem/QmodemPro (they both compute ZCRC incorrectly). But as of now Qodem can do Zmodem upload and download with rz/sz, Qmodem, QmodemPro, ZOC, and HyperTerminal.
August 31, 2007
Qodem can now send and receive to itself with no errors, which is a major victory for it. Now to test it against ZOC...
Testing against Hyperterm and ZOC, basic transfers work but there are some oddball corner cases, in particular Hyperterm's "Skip" button on receive sends a ZRPOS that equals file size which my code thought was a violation of the protocol and also (I think) a 1024-byte file that Qodem didn't end the transfer the right way with. I found both issues in the code and fixed them, and a bug in the input bytes buffer handling in qodem.c that left a bunch of garbage onscreen after a file transfer, and also the bug that entering DEBUG emulation didn't print the header right away. So now those are all fixed now in CVS.
August 30, 2007
Whew! All day today has been Zmodem vs lrzsz. I found another "bug" in the protocol: you can't actually escape out additional characters so it won't be possible to protect users from getting kicked out if they upload the rsh/ssh/rlogin/telnet disconnect sequences. I also found half a dozen bugs in Qodem. But it seems a lot more stable on uploads now, and uses a heck of a lot less CPU with the new streaming arrangement. I have null modem cable at work I'll be able to use to test Qodem vs Qodem on Zmodem uploads and downloads, and then I'll need to test against ZOC on Windows to see how it behaves.
I still have some "glitches" around the file handling as one user noticed, and I've got that whole Xmodem and Ymodem debugging cycle to get through (sigh). But I have hopes that when that is done I'll have the -G protocols and will cut a new release. Maybe (cross fingers) by Halloween.
June 29, 2007
Spent all day reorganizing the way file transfers move data in process_incoming_data(). Still working on getting XYZmodem working again...not fun. The lrzsz version of sx SUCKS to debug against.
October 11, 2006
Just found the bug that caused my TradeWars macros to crash: substitute_string() had all sorts of errors if it had to do more than 1 replacement. Fixed in CVS, will eventually roll out to 0.1.2.
July 20, 2006
Removed the last use of perror() and random printw().
Forked the code on my dev box to 0.1.x (patches) and main trunk. I'm not sure how to tag CVS appropriately, so we're stuck with the latest-and-greatest on Sourceforge.
July 19, 2006
I added a quick section in qodem.c to set ESCDELAY to 20 milliseconds (if it isn't already set by the user). This makes vi much more responsive. :)
Now I'm starting to encapsulate the zmodem code into 'qrzsz', a separate program to be a drop-in replacement for the public domain rzsz. I'm doing this because it will force me to clean-up the API to zmodem so I can more easily add it to other programs, and also to pave the way for the Unix HS/Link port.
July 15, 2006
I'm trying to find the "keyboard editor doesn't load keys" bug and am stymied so far. There! Got it, have to switch emulation, THEN hit Alt-J and see nothing. OR, connect localhost ANSI, then hit Alt-J. Looks like if I hit Alt-J from the first everything works OK.
Found it! I was calling reset_function_key_editor_textboxes() AFTER switch_current_keyboard(). I honestly don't know how it worked the first time. (I really need to re-factor switch_current_keyboard() someday.)
Packaging up for 0.1.1 release. Code size: 45237 lines.
July 13, 2005
Down to one major bug before the 0.1.1 release (unless another crops up of course). The disappearing cursor bug actually only appeared with the serial port stuff because all the other callers to notify_form() and notify_prompt_form() knew what to do afterwards. Nice.
July 12, 2005
I added a wrapper to beep() so that we can play Linux console music :-) . It also gives us the ability to later surface beeps through a real sound system if that's available.
Also, ANSI "animates" a little better at the price of performance. Online it looks fine, but local full-screen apps have noticeable lag drawing the screen. So I've exposed it as an option, with the default to NOT animate.
Three more critical defects to nail, then I'll probably cut 0.1.1.
I think I got the screen resizing problems fixed. There were a lot of small issues that arise when HEIGHT and WIDTH change suddenly, but now it behaves a lot like xterm which makes me happy.
July 5, 2005
Found rendering problems in Avatar and ANSI emulation thanks to Doghouse BBS. Avatar wasn't handling cursor position ( ^V ^H row col) correctly, and ANSI wasn't resetting cursor position (CSI u) correctly. With both fixes in, BRE looks good. :)
June 27, 2005
Just found an oversight: reset_emulation() wasn't being called until a connection was made, meaning the emulation-dependent variables in q_status weren't being set right off. It was exposed when I added the compose_key() function, which I'd kind of like someday to put everywhere in the program, so you could "compose key" a high-ASCII name into the phonebook for instance.
June 24, 2005
Tried that download again and landed on a "telnet>" prompt. Now THAT'S weird! I added escaping of "~" and "^]" to zmodem so maybe we won't accidentally do it during a file transfer, but it's very odd that I got the prompt at all.
I've added wrapping on column 80 to ANSI, Avatar, and TTY emulations, currently as an option in qodemrc. I might expose it as a toggle at some point.
Just fixed a slight rendering bug that was affecting my TradeWars 2002 game. :) Control characters that were meant to be glyphs weren't being printed.
June 23, 2005
Doh! A long Zmodem download from a Maximus board caused lots of strange behavior right as it ended. I think Qodem responded to something by trying to re-send the ZFIN, which did weird things to the BBS session. I'll have to run a download again in debug mode to see what I missed.
Just added a check to hangup so we're not always talking about a "modem". Now Qodem feels to me like a telnet/ssh client that can also do modem, not the other way around.
June 20, 2005
I put most of the important bugs/features on the Sourceforge bug tracker so any users I end up getting will feel more confident about reporting them. Also fixed the Avatar crash bug and made it so Qodem won't try to create directories just to emit a usage screen.
June 12, 2005
Got the modem to dial correctly and actually used a Maximus BBS! Woohoo! Exposed some problems with Avatar -- I don't yet support the commands in the May 1, 1989 addition to the AVT/0 text. Once I fix those I'll be ready to cut.
Ok, got most of them in. I'm going to pass on the two scroll region commands for this release because I haven't ever seen a BBS use them. :)
I've re-numbered the TODO to reflect a new roadmap. Basically, THIS release is the one I want to be known as "stable" for the next year or so, yet I'll continue making patches as serious bugs emerge. So 0.0.8 becomes 0.1.0 and the first patch will be 0.1.1 etc. It's a good release, way better than a lot of other 1.0's on Sourceforge and generally one that I think many people can enjoy. I'll also get an English-only version out in a few weeks so we don't have the LANG=en_US irritant factor all the time.
After I get the scripting and Kermit support in along with some other smaller features we'll have 1.0.0. That will be the "final" milestone, all the things I've personally wanted/needed in my very own terminal emulator. Other neat goodies and the Windows port are 2.0 and 3.0, respectively.
I just de-localized Qodem. It only took about 90 minutes, since I had an "un_gettextize" perl script that created a sed. Then hand editing the sed script, and viola! English-only Qodem that does NOT require GNU gettext to run! It's going online now.
June 7, 2005
I finally found the one extant copy of Sam H. Smith's farewell HS/Link post (SHSMITH.HTM) where he apparently has released the 1.21b7 source to the public domain. I've got a question out on Usenet about it, and I'll begin hosting the files myself on Sourceforge to see if anyone comes along with better information on the copyright status. But if it really is free I'm going to incorporate it and spinoff a command-line client for it so that we can at least one bidirectional protocol under GPL. (The Hydra protocol source code is not licensed in a GPL-compatible manner, even though its source is available.)
Also spent a couple hours poring over the last releases of a few DOS terminals. Only one feature is going to find its way into Qodem: "automated trigger macros" from ACECOMM. This would let a user define immediate actions based on the incoming data stream, which could make Qodem one hell of a MUD/MUSH client. :-) I'll spend some time crafting it appropriately for the Qodem interface, but I think it will do a lot to help users automate chores when online.
June 3, 2005
Screw RPM. Maybe it will re-appear in a future release, but for now I say the rpmbuild system needs to get easier to use before I want to spend any more hours on it. It took about two hours, total, to get qodem packaged as a Debian .deb package, because they use standard tools (ar, gzip) and have only an unusual filesystem layout to things. RPM has already taken two hours and I've got almost nothing to show for it. A .spec file that I can't use without creating $HOME/.rpmmacros because RPM wasn't designed from the start with a non-root user in mind.
So now we're down to only modem dialer before release.
June 2, 2005
Looks like Zmodem is basically complete now. It successfully does error recovery on fast modem links. However, uploads are slow and I know why: I'm not filling up the write buffer as fast as I could. Of course, this only affects very fast (as in local) transfers, but I'd like to eventually get it streaming full blast in both directions. Oh wait, testing it again and I see the problem: it's the disk access when DEBUG_ZMODEM is defined. Nevermind, we're seeing better than 1 megabyte/second on local transfers.
Now it's just testing the modem dialer and making an RPM target, then we release 0.0.8.
June 1, 2005
Got Zmodem receive to do error recovery, now trying to get send to do the same. Almost there, but it's pretty tricky.
May 23, 2005
I found a pretty serious bug in batch_entry_window() that would cause it to drop core if the screen size was too small (like under xterm). Now we ask xterm to resize to 80x25 if necessary, so that should improve things a bit. Also, backspace was very odd. I ended up querying the x location in the form so that I could ignore backspace if I was at the beginning of the field. Sadly, no one seems to be asking much about forms on Usenet.
Getting closer and closer to the 0.0.8 release. Right now it's just bugfixes and pounding on Zmodem.
May 17, 2005
Fixed the rendering bugs (cross fingers) in EDT and LSE. Not only was I ignoring a default value in decstbm(), I was mis-handling erasing lines containing characters with A_REVERSE flag set -- except that Linux console does that too and now it's expected behavior apparently. Terminfo calls it "back color erase" (bce). I'll probably need to do some work later to ensure it does what most users expect.
Did another survey of the terminal emulator round-up. Apparently there is a project on Sourceforge.net for the Linux console, but the code seems to be stagnant. I'm thinking (briefly) about asking if I can add VT52 support to it, and at least get it through vttest. OTOH it does handle LSE and EDT reasonably well, so it's not bad as a VT100-ish emulator.
Other projects look neat but mostly they are focusing on X11-based emulators. After our 0.0.11 beta release, I'm going to start advertising like crazy. I really want people to use THIS code to play their MUDs with. :-)
Long day. Heading home.
May 13, 2005
I finally tried to use EDT and LSE on VMS, and sure enough it broke my VT102. I'm going to debug that use case again and get it working before this next release.
May 4, 2005
Continuing in our run to a reasonable beta, I took out the strings that referred to the non-existent help system, so users shouldn't be trying to hit F1 and getting disappointed. Now I want to add the file transfer event logging, and we'll be almost done.
For this release, I'm going to package as .deb, .rpm, and Slackware-ish tarball in addition to the standard source tarball.
April 29, 2005
Removed an entire class of keyboard errors by wrapping most calls to wgetch() through my own function. It's nice because it makes the Linux console keyboard behave like an X (or DOS) keyboard: if you add META/ALT to a key within a form, it's processed just like Alt-keystroke, and most forms don't care about Alt. So if you hit 'Manual Dial' in the phonebook and hold down Alt you'll get the same output as if you didn't hold down Alt. Can't believe I waited two years to fix that.
I should be able to commit all this in about 14 hours. I'm going to wait another week or two before cutting the 0.0.8 release though, because I want to flush out any remaining memory leaks and bugs in the file transfer protocols. I want to really shoot for an almost-beta-quality release, something much better than my prior releases.
April 28, 2005
Slogging through the VT220 emulation now. Other than "protected areas (DECSCA)" the rest of it looks rather straightforward. Bringing VT220 into this release makes it the "big one", where there's enough function in one place to start using it for real work. The only features left after this are QSL scripting (huge work), Kermit, and a handful of small-to-medium features.
VT220 is a LOT more complicated than VT100 though. To really do it justice I should support a lot more character sets, which may mean mapping them to Linux's UTF8 or ISO-8859-1. I'm still thinking about that though, how far I want to take it.
Re-investigating the vt100.c and linux.c is good though. I removed a lot of unneeded VT52 mode checks. I probably have quite a bit more junk in linux.c that could be tossed.
Well, VT220 is almost passable at this point. Emacs, joe, and mc are all happy, and most of vttest is too. But I wish I could do some of the features it's got like soft fonts and user-defined function keys. OK, I could do the function keys, but I think that would actually hinder users who get used to Qodem-style keyboard macros.
I should be back on land in about 48 hours, then I can commit this in and mothball it until I'm ready to implement the scripting.
Code size: 43457 lines.
Sleep time. G-night.
April 27, 2005
OUCH! I just ran both Qmodem(tm) 5.0 and QmodemPro(tm) for DOS 1.53 through vttest. OUCH! AAAAH! I now see why the Unix gurus in the early 90's complained so loudly. Qmodem(tm)'s vt100 emulation is the worst I've personally run through vttest; even worse, development for QmodemPro(tm) had 2 years to fix it, and they didn't do a damn thing! Re-implement most of the UI in TurboVision, sure; but fix the vt100 emulation to be usable, no way!
I'm embarrassed. Royally. I loved that program enough to clone it -- going on 1.1 megabytes of source now -- but that's pathetic. I wouldn't be surprised if that's a large reason it couldn't stay competitive with Procomm. They cut too many corners. Or maybe people abandoned them because they flat-out lied:
"QMODEM 4.6 TEST DRIVE supports the standard VT100 functions; half/full duplex, set/reset modes, scrolling region, keypad application mode, special graphics character set, US & UK character sets, full display attributes including ANSI color extensions, programmable tabs and cursor control."
Bullshit. Scrolling region: broken. Keypad: broken. UK character set: broken. C0/C1 control codes: broken.
HyperTerminal(tm) almost gets it right, even does double-width and double-height. PuTTY's pretty darn close too (only one screen looked off). minicom's is mostly OK for real-world use (VT52 broken though). But Qmodem(tm)... it's real bad.
But you know what? Qodem mostly gets it right. I re-ran the tests with line wrap both enabled and disabled: no change in screen results.
HAPPY 2ND BIRTHDAY QODEM.
April 26, 2005
I now have technically all the features needed for the 0.0.8 release. I now need to spend a lot of time fixing the memory leaks and testing the file transfer protocols on fast noisy lines. If I get more time on this trip I may bring the file post-processing feature into this release.
Fixed some amusing bugs in music.c around the handling of '>' and '<'. Fixed those, then I noticed that ZCRC can actually do a real crash recovery (it takes filesize as an argument). lsz.c was too spaghetti for me to see it, but lrz.c sets it up for that; so I changed zmodem and now GC is saying that something (length 5) is getting smashed. (Maybe it's the ZCHALLENGE value?) Something else in the "fix before beta" category.
April 25, 2005
I just changed the splash screen (and fixed a bug in it). The old one looks "better", but it's just too close to Qmodem(tm) 5.0 to keep as is.
I finally had a chance to run Intellicomm for the first time. sigh I'm really sad now. The author put SO MUCH work into Intellicomm, and if I had had it circa 1990 I would have been lucky. It was designed to automate 90% of the drudgery that was a BBS: get new files list, exchange QWK messages, check email, store time in time bank, etc. You could set it to do all of the automation in the call and then beep you when done so you could play the door games. I would have had a lot more time in the evenings after school, and I think a lot more fun. "Watching TV.... Hmm, computer is beeping. Oh! Time for TradeWars." But then the Neverending September rolled through and all those functions are no longer required by practically everybody. File and message BBSes (particularly the DOS- and OS/2-based ones: WWIV, Telegard/TAG/Renegade, PC-Board, RBBS, Maximus, etc.) are almost extinct now that we've got WWW and FTP everywhere. It's really disheartening in a way. I only hope he got lots of registrations to pay for it.
But now that I've seen "keyboard repeat buffer", I think I'll skip it as a feature. Modern OS's have shells with history and readline.
Got most of dialing working now. I won't be able to test the connect part until next week (life offshore, oh well). Made the phonebook recognize the difference between a phone # and a network address. Got manual dial "working" (as best it can without a real phone line :) ). Just hope the boat doesn't go down, or I'll have a bitch of a time re-doing all this. Code size: 41981 lines.
But you know what? I don't need minicom anymore. WOOHOOO!
April 24, 2005
Got minicom-style lockfiles and serial speeds up to 115200. Don't have 230400 yet, but neither does minicom on my hardware, so it might be kernel driver or setserial.
Gonna be a pain adding the serial parameters to the phonebook. Sigh. I'm glad I waited until Qodem was functional in most other respects before heading into the serial port stuff, I think I would have lost patience.
I can talk to a modem now. Just need to put the stuff into the phonebook and then get dialing working and we'll be set. Phonebook now has the ability to set port settings.
Just realized that the modem is going to introduce some subtle problems. For instance: if I have a phone BBS and an SSH host both selected in the phonebook, which one should I "dial"? If I want to integrate the SSH seamlessly in then I need to detect the lack of password as a "busy signal" and keep dialing. OTOH I can put up a message saying that automatic dial might not work well with a non-serial connection tagged and ask the user if they want to remove the non-serials from the list.
Taking a break, I realize that my initial impulse behind this project has proven to be true: that we can make a decent terminal emulator a hell of a lot faster with recent software tools than they could back in the 80's. Sounds like "duh", but seriously: curses handles the keyboard about 95% right and the screen about 100% right. (I'm quite impressed that curses only emits what has to be changed on each call to refresh().) POSIX I/O makes moving bytes between pty's, serial, and the console a breeze, relatively speaking. By the time I get home I'll probably have a reasonable beta (except for scripting) that could compete with the original Qmodem(tm). That's in only about 4 months of full-time coding (spaced out over two years in offtime).
Damn, long two days. Code size: 40972 lines.
April 23, 2005
Added the communications parameters form. Now to get control of the serial port...
This is a huge break-off from Qmodem(tm) by the way. Qmodem(tm) was serial-only (+ telnet only in QmodemPro 2.x for Windows). I need to make sure the serial port behaves in a reasonable way. Should it automatically open on startup, or only if we "dial" a serial number? Should it close right after the call completes, or should it wait for an explicit command? When the user uses a non-modem call, should it also unlock for other applications?
Most programs just open the serial port and assume it's theirs, or they take the other extreme and don't talk to it at all unless they need it to dial out. I want to emulate a real DOS terminal though -- where the serial port is generally open unless it doesn't have to be. Maybe.
April 22, 2005
Well, work has sent me offshore. This should bode well for the 0.0.8 release, only because outside of my mandated 12-hour days I've got nothing else to do. Yippee.
Well, I've got the modem configuration menu stubbed in. Now to get the serial port stuff functional... It'll take a day to get it right. I've peeked a bit at minicom (thanks to the GPL) and see I'll have to do mark and space parity at the application layer.
April 21, 2005
I can't quite get my head around the "Keystroke/Command Buffer" feature, I think it's going to be removed. I first saw it advertised in the README.1ST file distributed with Intellicomm 2.01 and it sounded neat, particularly if you need to configure a Cisco router, but I can't quite visualize a good way to do the user interface for it. I suppose I could run Intellicomm and see what they do, but for now I'm just going to leave it be. I've got keyboard macros, split-screen, and QuickLearn coming in that will probably cover most people's itch I think.
If anyone out there knows how they would want to "edit typing errors and/or re-send any of the last five BBS commands entered" please let me know.
Fixed the leaks in keyboard.c and options.c. Pretty silly stuff, but easy to see how I lost track of it. I am so glad the Hans-Boehm garbage collector is out there.
April 20, 2005
Sheesh. Tired. Work took a lot out of me.
I'm starting to see the light at the end of the tunnel on the first real release. Once I get serial port and scripting support in, I'm going to spend a long time exercising features and then cut a public beta. From that point I'll see about packaging for distros and ports to other Unix-like platforms. It'll be neat though when it happens.
I took a peek at some of the competition: minicom. The "bad news" for me is that minicom is really small and a good mix of features for most people. It doesn't actually uses (n)curses so it doesn't have as much "bloat" as Qodem. However, its VT102 emulation is poor, even compared to the raw Linux console. (To be fair, it was a whole lot better than Qmodem(tm) and most other DOS-based emulators.) I'm even more proud now of the work I put into beating the vttest scenarios. Specifically, I was looking for the /var/lock/LCK..ttySx code -- I'm probably going to do a more-or-less direct rip (with attribution of course) so that we can share a modem properly with pppd.
April 14, 2005
Send is now working. All I need to do now is add timeout checks and do a walkthru before 0.0.7 goes out. I'm hoping that's TODAY. :)
I got ZTerm to transfer a 1.5 meg file through a 3-wire serial link (using a small Perl program to shunt characters from STDIN to /dev/ttyS0). I'm reasonably confident now that my implementation will talk to Zmodem's-other-than-lrzsz.
Sigh. There's GNU gettext FUCKING UP MY FUCKING TRANSLATION FILE AGAIN. This time it was 'make dist' that corrupted en.po. I swear the gettext "tool" chain is going away before I go to "production/stable".
Released. Code size: 37501 lines.
I'm getting food now.
April 13, 2005
Zmodem send is really pissing me off. It looks like rz has bugs in its CRC32 stuff. For example, it's trying to compute the CRC on a binary header with 8 input bytes, and of course I've no idea just where those 8 bytes are supposed to come from (since only 5 are used by sz). When I switch to hex headers with 32-bit "sub-packets", I get other problems. This is really irritating.
Yup, it IS a bug in rz. Namely, rz ASSUMES a CRC32 on ZSINIT even if the header is CRC16. Blech.
Using the weird negate-the-crc-function-output I used for ZCRC and suddenly it works again. This is really f'ed up.
April 11, 2005
Receive is now as complete is it's likely to ever get, barring bugs. Stubbing in send, and a few minor cleanups (don't allow non-supported protocols to be selected, etc.).
April 8, 2005
Receive is now mostly complete. I expect some most corner cases to pop up, particularly in the almost-never-used options (ZCOMMAND et al), but I can cd into a directory now and say "sz *" and everything comes down, with crash recovery and skipping already-downloaded files. Send should be significantly easier than receive now that I know what to put on the wire.
I cleaned up the Alt-Z help menu (alphabetized each section) and put in a system call to fire up the editor on Alt-N configuration. Much nicer than hand-editing IMHO. Next week I hope to complete a passable Zmodem send and then cut for release.
April 6, 2005
Work took some time today, but I managed to get receive ZSKIP mostly functional.
April 5, 2005
Wrapping up Zmodem receive. ZCRC handling was a bitch to debug though. I think the CRC32 function I've got doesn't match the one in lrzsz -- like it does some weird negation thing in ZDATA that is correct but is supposed to be handled at my layer, but then doesn't do it for ZCRC. Or maybe ZCRC is just not well supported anywhere. What I find hilarious is the fact lrz doesn't even implement ZCHALLENGE even though it's part of the "spec". But anyway, now we're using CRC32 by default and only three features away from a most decent receive (ZSKIP, ZRPOS, and error handling).
April 4, 2005
Zmodem receive is working in a very rudimentary way. Need to fill in a lot of gaps still...but my writer's block is finally over!
That's it for today. We can receive multiple files in sequence, so long as the link is reliable already (i.e. TCP). Still need to do crash recovery better and get all the small breaks closed -- it's way too easy to make me drop core right now. But hey, receive is the hard part, send will be a snap after this.
April 1, 2005
Progress! I finally figured out why I hated my original Zmodem code. It was a direct rip of the X/Ymodem stuff that had this godawful huge state machine with gobs of copied code between sections. I just now refactored zmodem_receive() to call sub-functions for each major state and poof! instant readability improvement. Just gotta get the decode bytes function to handle the crc part and we'll almost be back up to receiving files. OTOH I'm moving and it may be a few more months....
March 8, 2005
I might get some time this spring to code on this project, at least I hope so. I've been using Qodem at work for the scrollback and keyboard macros functionality, it's been nice.
I finally outlined what I wanted the scripting language to look like, after looking at Procomm, IntelliCom, Telemate, Terminate, Kermit, Commo, and others I forgot. And Qmodem(tm) of course. It'll resemble the most-critical subset of C that I use myself, but won't have pointers or direct memory access. However, it will have one kind of reference (structref) so I can add all the typical data structures necessary for complex scripting. The VM is going to be very trivial and stack-based; we won't be winning any speed contests with it.
January 1, 2005
AAAAARRRRGH. Zmodem is pissing me off. My main complaint is that Mr Forsberg had no idea what he was doing when he put together the "protocol". Basically, here's what I don't like about it:
The CAN escape character is used both for demarking data boundaries (CRC escape) and encoding field data (control character escape). Hence I cannot run all of the input data through a decode function and then parse out fields. Control character escaping is done inside the CRC check and header arguments too, so I have to make this fugly hack job.
The CRC escape sequence uses the byte within the escape sequence as part of the calculated CRC value. So when a file block comes in I have to check the CRC and then magically remember to ignore the last byte when I write data to disk.
The Unix file permissions mask included in the ZFILE data is not encoded in a platform-independent manner so I can't use it.
He didn't bother documenting the fact that he was transmitting the number of files in the batch and the total filesize in the batch, so I can't use those either.
ZCOMMAND! What the hell was he smoking allowing one end of the link to issue shell commands to the other side?
I sincerely hope that when I get to implementing Kermit I'll be able to decode the data all-at-once before parsing for header information, and that the CRC calculation doesn't require mucking with the file data.
So I'm about 20% done with Zmodem, probably 50% done with receive. I need to add a static buffer that I copy the incoming bytes to so I can ensure each major stage has all the data it needs before I start decoding things. Then get the handshaking right at the end of file and add support for ZCHALLENGE and recovery.
THEN I need to get the send side working, and THEN I have to get these options exposed in qodemrc. After all that I'll cut 0.0.7.
December 7, 2004
Found some redraw issues surrounding save_form() and view_directory(). It's ugly...I may end up redoing the forms in a future version because they don't know enough about how to redraw themselves over each other.
December 6, 2004
ANSI Music is now in. What an abomination!
Almost there though...just Zmodem remaining in 0.0.7.
December 5, 2004
ANSI Music is about 60% stubbed in now. I can play tones through the Linux console, and I can detect and partially parse a GWBASIC PLAY string, but I still don't have the last bit done.
I've found a number of small usability defects in the console (scrollback buffer handling, VT100, line wrap, etc.) I now "eat my own dog food" and use Qodem as my primary interface at work. I think this next release will really be a good one for most people. I still need Zmodem finished out before I can cut it though. Work is sending me on a trip and I might find some time to get it done then.
October 13, 2004
I'm putting in support for "ANSI music", a serious misnomer for a PC-ROM BASIC hack to the ANSI.SYS driver. I've found a handful of other tiny things to fix too before we go out on 0.0.7.
October 9, 2004
Stubbing in the Zmodem FSM now. The last time I looked at zmodem.c I was buried in a mess of Xmodem and Ymodem implementation junk (which still looks NASTY as hell) and was also using the lrzsz output parsing junk -- that was back in July. Now I'm going for a real Zmodem implementation to get the block size and stuff right.
September 30, 2004
Finished up the ASCII transfers (CR/LF handling). Also added a flag to automatically enable doorway mode on connect -- really handy when you connect to Unix boxes and want PgUp/PgDn to work right off.
September 29, 2004
Got ASCII upload and download working, but discovered some interesting "problems" with the pty master/slave. I switched over to forkpty() instead of the ripped-from-Eterm code and it behaves better by a bit. However:
The default behavior of non-blocking causes write(q_child_tty_fd) to hang indefinitely if the program on the other side isn't actually reading. I had to make it non-blocking so the user can retain control and cancel the ASCII upload.
vi sucks at reading fast, hanging if the buffer fills too quickly. cat > filename works much better.
So now to plug in the ASCII translate table functions and ASCII upload/download will be complete. Got the tables in, now to commit changes...
Figured out how to get ssh to set the right lines and columns when it connects to a remote system: TIOCGWINSZ. ('strace stty rows 100'). So that's fixed. Also figured out why arrows weren't making it through in Linux emulation, so that's fixed. Commit and hang with the wife now... ;-D
September 28, 2004
Figured out how to auto-detect the Linux console. ioctl(fd, GIO_FONT, ...) will return 0 only on a VGA VT; it returns EINVAL when run against a PTY. So now we always look our best (cross fingers). :-)
September 27, 2004
I stubbed in the ASCII translation table editor. Now to plug it into ASCII upload/download....
September 26, 2004
I got a full-time job in mid-July and that has taken all my time. Today is the first day I actually looked at the Qodem code and got it to compile again. I'm going to sweep through some old bugs for these next few days and then work on Zmodem after.
Just fixed a bug in the keyboard file handling, and added ANSI fallback support to Avatar emulation.
Added wildcard filter support to view_directory(). Committing it in.
July 1, 2004
Well, I'm back in Texas and resuming work on this project. I just ripped out the code that glues Zmodem to lrzsz, so now Zmodem is broken until I implement it. It's been a very slow morning, for some reason I've just been DRAGGING along this week. I think I'm going to take a nap and then start into the guts of real Zmodem.
May 29, 2004
That's it! We've got a couple bugs to iron out and I'll just let it lie for now. You can load up default.key, put in $USERNAME^M and $PASSWORD^M into a key, and then use that key on sites connected to via phonebook.
Committing it in... We are now mothballed until I finish the move to Texas and get settled in to a new routine.
For my future reference here is the 'wc' output:
1449 4656 34834 ansi.c 47 244 1675 ansi.h 419 1229 9994 avatar.c 45 235 1623 avatar.h 579 5566 32498 colors.c 126 619 4624 colors.h 362 1769 10456 config.h 1380 4004 38882 console.c 57 289 2136 console.h 214 799 5876 debug.c 46 238 1653 debug.h 264 819 7019 dialer.c 45 235 1607 dialer.h 619 1776 14680 emulation.c 120 505 3915 emulation.h 1105 3790 34087 forms.c 60 325 2241 forms.h 188 656 4550 getopt1.c 1055 4477 30212 getopt.c 169 903 5861 getopt.h 3440 8920 84533 keyboard.c 49 245 1787 keyboard.h 3680 11363 87768 linux.c 68 414 2727 linux.h 437 1497 13388 options.c 76 288 2341 options.h 2734 8119 76040 phonebook.c 105 393 3021 phonebook.h 1688 5442 53558 protocols.c 158 710 5556 protocols.h 1122 4008 32310 qodem.c 293 1046 7550 qodem.h 92 362 2689 screensaver.c 48 249 1757 screensaver.h 1634 5829 48005 scrollback.c 99 485 3855 scrollback.h 286 727 6825 states.c 91 428 3212 states.h 143 672 4472 status.h 6 14 92 test.c 138 686 4457 tty.c 43 227 1513 tty.h 3367 10261 80211 vt100.c 56 298 2058 vt100.h 548 1804 14542 vt52.c 49 256 1765 vt52.h 2270 8264 63144 xmodem.c 63 309 2147 xmodem.h 31132 106450 859746 total
May 28, 2004
Wow, four whole days with no entry. I've been slowly working through the function key editor. Sucks that I've got to do every small operation for about 75 keys, but OTOH C macros makes it much easier: I've got about six functions that are just 1 macro for each key (copy to/from disk, copy to/from editor, load terminfo keys, etc). What's left is the actual using of the bound keystrokes in keyboard_handler(). That should be a rather quick one, and then a sweep for bugs. I notice that aborting an edit still leaves stale data in the textboxes on the next load.
And once I have F9==username^M and F10==password^M working, THAT'S IT until the end of the summer. (BTW I purchased a copy of Tim Kientzle's The Working Programmer's Guide to Serial Protocols and hope on the next development push to get Zmodem and Kermit working. To Mr. Kientzle: I intend to write my own code rather than incorporate the code provided with the book.)
May 24, 2004
I've been a little lax on the worklog, but I've actually gotten more code working. Ymodem receive works now, but I notice 'sb' will use 128-byte blocks unless you explicitly ask for 1k blocks, AND 'sb' puts an extraneous linefeed in the terminator block that's not part of the spec so you always have one error count before the end.
Today I need to do some fixing up on the strings in xmodem.c and also do dynamic re-calculation of the # of blocks when a 1k protocol sees a 128-byte block. Got the protocol cleaned up, I think it's fully working now.
Added terminfo support for unknown key presses -- should be very handy for ANSI mode because everyone disagrees what ANSI really means. (I actually saw a BBS send DECAWM! I mean that's pretty ridiculous, DECAWM is not at all in ANSI.SYS.)
Just added commandline support, so you can say "qodem ssh hostname" or "qodem csh" and get a shell inside the qodem environment. Cool.
But ssh isn't passing LINES and COLUMNS correctly. Wonder why? I just saw the -e none option to eliminate the escape character ~.
May 22, 2004
I'm re-factoring xmodem.c already. I don't like the way it morphed into spaghetti code once I added all the cases for vanilla, CRC, 1K, 1K/G, Ymodem, and Ymogem/G! Timeouts aren't being checked in a straightforward manner, it's manipulating q_transfer_stats directly, etc. Ugh.
At least I know HOW the protocols work now. I'm fighting my way through an Atkins diet weekend so it's slow going. (Any of you who have actually been on Atkins or something similar (Atkins is only the third person to discover that all-meat diets work) know the feeling of tiredness that comes after being in it for a few weeks.)
May 21, 2004
Ymodem send is sorta working. I'm cleaning up small glitches in the batch upload progress window. YMODEM.DOC, the "definitive" source for the protocol, is ... ahem ... a bit lax. Other people who have implemented their own have complained on Usenet about it, only to be responded with "why don't you just se DSZ.EXE?" sigh I'm starting to call some of this work "digital archeology" because I have to determine what the designers intended by looking at the existing artifacts rather than the specifications, and also (more importantly) because even though we depend on XYZmodem almost daily there are NO BOOKS whatsoever in the libraries and bookstores that really discuss it. Hundreds of books on signalling, AppleTalk, OSI stack (which no one actually uses), BGP/RIP routing protocols, but nothing on the good old reliable file transfer protocols.
May 20, 2004
I've got an interesting crash. The _() macro (gettext()) is hanging indefinitely with a smashed stack, after I've done a Xmodem-1K download. Ooh, actually it's after ANY kind of Xmodem download. This is good, believe it or not, because I can isolate it to xmodem_receive(). Still, it's a very odd item... Let's rebuild and see if it remains...still there. But at least it's a nice reproducible bug. And it's not there when the transfer is aborted. Got it! I was fclose()'ing the same file handle twice.
Got Xmodem send mostly working, at least in Xmodem normal and CRC mode. I'm sure there's going to be a whole cycle for testing it against various BBSes and versions of rx/sx before I can say they work 100%.
Now for Ymodem. Same idea, just bigger. Probably deal with it tomorrow though. Since 0.0.6 I'm trying to keep an even keel on things. My cat needs some more attention. :)
May 18, 2004
0.0.6 is out!
May 17, 2004
Got the phonebook Find Text and Find Again functions working. Zmodem autostart also works. I tested VT102 against flash.c and resetting the emulation and clearing the screen corrects the display.
It's starting to feel like a very usable program now. I can use vi and joe from inside a local ANSI shell; the scrollback sometimes is glitchy after some direct cursor addressing commands, but the visible screen is 100% correct on my particular test load of mc, emacs, joe, vi, and bash shell. I think with 0.0.6 I'll be ready to actually eat my own dog food as they say.
Heehee. I just scp'd rz to a box, then ssh'd over to it, and sent sz over Zmodem and finally used the remote sz to send rz back to me. And it all worked.
(Doh! Spoke too soon. Emacs + ANSI = color problems. Again.)
I greyed out the 3 protocols and 1 emulation I don't yet support. You can select them but who knows what will happen. grin I just have phonebook Undo function and Xmodem/Ymodem to verify and then I'll seriously be ready to cut.
May 16, 2004
Whew! (I say that a lot lately. :) ) I telnet'ed over to a couple live BBSes to test the ANSI support and found a number of minor bugs to fix. But I slogged through it, and noticed a few oddities out there:
CSI Pn L (ANSI X3.64 Insert Line) is being used by Emacs with no parameter. CSI Pn M (ANSI X3.64 Delete Line) is being used by Emacs with no parameter. CSI Pn @ (ANSI X3.64 Insert Character) is being used by lynx. CSI Pn P (ANSI X3.64 Delete Character) is being used by lynx. CSI Pn G (ANSI X3.64 CHA) is being used by BBSes. CSI Pn d (ANSI X3.64 VPA) is being used by BBSes. CSI Pn b (ANSI X3.64 REP) is being used by lynx.
CSI ! _ is being used by BBSes to detect RIPscrip.
CSI Pr ; Pc f (ANSI X3.64 HVP) is seeing Pr=0 and Pc=0. The standard specifies top-left as (1, 1).
So I got those added to ansi.c, along with a generic control character handler for ANSI and AVATAR. Then I tested VT100 and LINUX against mc, lynx, and emacs and saw that Insertion/Replacement mode had a minor bug and fixed that. I think we've got all the emulators running correctly now. (Certainly better than the DOS versions of Qmodem(tm) and QmodemPro(tm).)
I'm going to divert tomorrow to phonebook stuff and finally get down to the protocols. I think before this is over I'm going to have to re-implement the protocols myself. lrzsz's code is pretty much spaghetti on the inside (even the last maintainer commented on that inside the source), and Qodem is completely dependent on the English-only output of it. I see no need to implement CompuServe B+, but IND$FILE might be handy later down the road.
Well, more immediately: I need to get about five features in the phonebook, then verify AVATAR and get Xmodem and Ymodem working through lrzsz. When those are done -- probably three more days -- I'm going to wrap 0.0.6.
The version beyond that is going to be focused on wrapping the keyboard mapping functions, stubbing in some scripting hooks, and VT220. I don't expect to get started on that version until the end of the summer: we're moving across the country and I need to get a new job. So this week is my great push for a reasonably stable version I can be proud of showing off.
[LATER] I just got two features in the phonebook: passwords are covered up with *'s so casual onlookers won't see them, and the PgUp/PgDn/Home/End navigation keys are working. (And there was a serious bug: the phonebook couldn't handle multiple pages of entries.)
Finally, I noticed that all the recent activity has pushed Qodem up to the second page on Sourceforge for projects in the "Terminal Emulators" category. Go me! :)
Sleepy time. It's only 12:00am too!
May 15, 2004 pt 2
Whew. Finally got Linux function keys and colors working alright. 'mc' looks nice, but I still have a few ANSI cursor movement commands to implement before it's really sharp. [LATER] Got it. Now I can run Qodem from inside itself and still get colors. However, ACS_?ARROW characters don't render for some reason.
Next up: protocols. I really need to debug what's going on with Zmodem, and then get autostart working, and finally get Xmodem and Ymodem up and running. It's pretty bad right now, this is rather old (for this project) code that never got a decent sweep.
[REALLY LATER] Well, I slogged through the first compile on Solaris. Yech. All I can say is at least it isn't AIX. The vt100 term doesn't have all of the graphics glyphs defined, so windows have top and right side and letters elsewhere. Whatever, it's 2:10am and I'm going to sleep now.
May 15, 2004
Again, very sleepy: 1:44am. But I've been on a rather good roll so far. I had this cool idea for DEBUG emulation and got that working in only about an hour. Then I found some books circa 1985 from the library that talked about "teleprocessing terminals" (TTY) and "intelligent terminals" (VT52, IBM 3270). Heehee. It's so easy to forget these days how much really ran on those machines. I remember old Columbo mysteries where some of the "computers" didn't have monitors.
But I recalled from the book that TTY is the evolved form of Baudot, and those control codes were actually a semi-official standard before ANSI X3.64 arrived. So I beefed up tty() to properly handle the control codes. Then I remembered what it was like to use a real typewriter, so for sheer perversity's sake I added underline support. I'm actually a bit proud of the fact that my "dumb terminal" will actually behave like a REAL non-intelligent terminal from the 1970's would.
A few other nice touches: up-and-down arrow while browsing the scrollback; ENQ support on all emulations.
Tomorrow ("today" after I sleep) I'm going to add color to the LINUX emulation, and then add the application/numeric key bindings for VT52 and VT100/LINUX.
May 14, 2004
Got my first response to a query on comp.terminals! I've added some emulations on the TODO. SCOANSI looks like it'll be a good challenge...especially on the keyboard mapping.
It's been a busy night. I've got Linux console emulation separated out, tomorrow I'll add the color changes. VT52 is now a "real" VT52 as in it's very limited, but I added ANSI colors (parameterized) because 'ls' puts the colors out even though the termcap clearly does not have them.
Sigh. I'm quite sleepy. I'll finish this update tomorrow.
May 12, 2004
What in the hell is a "fuzzy translation"? It all started with a debian utilities update and FOUR HOURS LATER I could finally compile the project again. I love GNU but DAMNIT this was a FUCKING WASTED DAY because they stopped supporting their old way of doing things. Naturally they say "read the documentation" except that the docs don't have a section called "What we recently changed that probably broke everything you're doing."
Then when it compiled again I was trying to see the unknown keycodes on the numeric pad and the translated message wasn't being used! Huh? More digging and I discovered the following:
msgmerge marks entries 'fuzzy' for no apparent reason. Then these entries are NOT ACTUALLY USED at runtime! What the hell is that about? I can't get msgmerge to really believe that a given string is the right translation.
msgmerge marks entries 'obsolete' that aren't directly visibly accessed in the code. Guess what? That means you can't have a programmatic key string to do the lookup at the appropriate time later or in a loop (like I do in options.c).
I am very seriously considering ripping out all the GNU build utils and returning to a simple Makefile. This is pissing me off like you wouldn't believe. I keep running into these productivity traps and the documentation just gushes about how wonderful the "GNU Way" makes life easier.
You know what makes life easy? Simplifying the project down so that very little platform-specific code is in it, a 'make what' shows the platforms supported and 'make XXXX' builds it.
Nah, I can keep this problem isolated to the po/ Makefile's. It's just msgformat that is screwing up the en.po before it ever installs.
May 11, 2004
Version 0.0.5 is now released!
May 10, 2004
I was just going to take a "break" from the hard VT100 stuff for a slick-ism: when run from a Linux console Qodem uses VGA glyphs, but when run anywhere else it uses ACS_* characters. It's very subtle but damn nice: "correct" drawing characters around the menus and such.
A "side effect" of this work is that the VT100 emulation now supports multiple character sets. :-) So though I didn't plan it at all, we've now got the most critical 2/3 of the special graphics character set available, AND we have the British pound sterling symbol in the right spot.
Quick line count recap:
658 1869 15007 ansi.c 45 235 1609 ansi.h 299 868 6811 avatar.c 45 235 1623 avatar.h 578 5564 32478 colors.c 126 622 4638 colors.h 276 1353 7837 config.h 1232 3588 35375 console.c 54 277 2050 console.h 252 805 6742 dialer.c 45 235 1607 dialer.h 369 1169 9947 emulation.c 103 446 3394 emulation.h 1093 3761 34846 forms.c 55 296 2045 forms.h 374 833 6909 keyboard.c 42 225 1537 keyboard.h 413 1404 12433 options.c 72 284 2245 options.h 2064 6339 59978 phonebook.c 99 370 2802 phonebook.h 1730 5880 55318 protocols.c 109 477 3663 protocols.h 923 3135 25320 qodem.c 210 772 5532 qodem.h 93 364 2709 screensaver.c 48 249 1757 screensaver.h 1544 5460 45335 scrollback.c 98 477 3732 scrollback.h 270 692 6540 states.c 88 415 3111 states.h 123 580 3883 status.h 124 635 4150 tty.c 43 227 1513 tty.h 3071 9278 71569 vt100.c 51 267 1828 vt100.h 464 1245 10594 vt52.c 45 235 1609 vt52.h 17328 61166 500076 total
VT100 is getting a bit large, but it effectively does everything now except double-sized characters (never) and VT102 insert/delete character/line codes.
There's a serious bug in the VT52 sub-mode that broke the test, but I haven't dug into it. I had a chemical engineering final exam this morning that took precedence. grin
May 9, 2004 part 2
Oops. Just broke the VT52 test when adding origin mode and scrolling region stuff. Probably doing something wrong in the scroll_up/scroll_down code in scrollback.c....
Well, I'm getting really darn close to wrapping 0.0.5. I've just got to fix the VT52 tests, then add support for the VT102 insert/delete codes. I've got plenty other things I'd like to get in, but I'm already better than QmodemPro(tm) was. I still need to test the VT100 and ANSI support with mc, joe, emacs, vi, and lynx. Once those are all happy with both emulations I'll call emulation work done until 0.1.0.
Then comes (whew!) the Zmodem tests across long connection chains. Once I've got that working -- which should just be figuring out the command-line flags to pass to it -- the I'll close on 0.0.5 and start fleshing out the remaining phonebook functions. I'm hoping to get past the VT100/ANSI and Zmodem stuff in the next two weeks.
It's been fun though getting this working again.
May 9, 2004
Wow, look at that. Five solid months of inactivity. Ah well, we made some good progress today. We've got cursor positioning, VT52 compatibility, and terminal reports supported in VTTEST. The really big kahuna still remains: "screen features". And I'm definitely going to have to make some compromises on that one. There's just no way to draw the graphics glyphs with a curses-based program, nor is there "smooth scroll". So I'll probably have to detect and "box" out the glyphs that won't render correctly and prevent codepage changes. Small price I suppose for my target audience: they should still have good emulation on VMS systems and over ssh/telnet to typical terminfo/termcap'd applications. I'll just make a note on it as a known limitation and recommend xterm for those who really need accurate-looking glyphs.
I still want to add a layer of codepage translation for Qodem itself, so that for instance it can detect that it's running on a non-raw console and then use the curses ACS_* characters instead of IBM VGA glyphs. Ah well, I've been coding pretty much 100% for the last seven hours, time for sleep.
January 1, 2004
First day of the new year. My resolutions to the wife are 1) to be busy, and 2) to stop buying shit from eBay. :)
I had surgery December 19, and got laid off December 23 (it was announced November 19). So now I've got more time to put into real code and not the bullshit pay-bills code. Yay.
I could rant a bit about how bad corporate code is, but alas I can't give any details because I signed a non-disclosure agreement about my slave^H^H^H^H^Hemployment with IBM. (I AM allowed to say I worked for IBM, but I can't name any customers who bought the junk, what I did specifically for the company, or exactly who inside the organization is running it into the ground.) Fortunately my open-source work is nowhere near the domain that my one-year non-compete clause covers ("autonomic computing" and/or "web site analytics").
Regardless, I'm coming back up the curve to this code. I fixed the tabulation set/clear so that vttest passes. I'm still working on the big kahuna "cursor movement" test that even Linux console can't fully pass.
November 16, 2003
vt100.c is now the largest source file. Go figure. :)
November 14, 2003
I got most of the VT100 functions stubbed into the state machine. The remaining functions are tab stops (horizontal and vertical) and scrolling regions. I just re-factored the functions into real functions and suddenly VT100 doesn't look so bad. Complex for sure, but thanks to http://vt100.net it's not too crazy.
In other news, I'm stuck at a meeting at work. It's VERY boring, corporate junk, the kind of thing that naturally happens when managers intersect programmers. They think that a new process will make up for passionless clock-watchers who just happen to know a little Java. For anyone actually reading this, stay away from anything with "Tivoli" in its name. 'Nuff said.
October 31, 2003
I finally fixed a stupid bug in cursor_right() that made 'mc' look like crap under ANSI emulation. Now mc looks good, so we've just about got ANSI beyond what ANSI.SYS could do, so it should be suitable for BBSes.
I also put the skeleton in for the VT100 state machine. It's going to take a while to get right, and the testing effort will probably be a bitch, but by God we're gonna have a solid VT100 before I switch the project release state to alpha.
October 29, 2003
Been culling small bugs, it's very fun again. I verified that uploads and downloads over ssh are working. Once I get the vt100 through vttest and cut 0.0.5, I'll have matched the critical features of the zssh project...that'll be cool. I'm sure zssh has better scripting support right now, but that'll come to qodem eventually.
October 20, 2003
Just cut 0.0.4! Wee!
Integrated with the Hans Boehm garbage collector. Pushed the remaining phonebook stuff to 0.0.6. Now it's vt100 and vttest.
October 16, 2003
Added the logic in for Q_STATE_UPLOAD and Q_STATE_BATCH_UPLOAD. No idea if it works yet of course. :)
My Emacs learning curve is mostly complete. Lots more Ctrl keys to mess with, but I do like the syntax highlighting and parenthesis matching.
October 9, 2003
Got batch_entry_window() mostly stubbed in. The next major piece is actually batch uploading.
October 6, 2003
Fixed view_directory() finally. That's an important screen before the batch upload window.
October 5, 2003
I finally had time to write some non-Java code in a while. Life's been busy as hell these last few months...got married August 1, took three trips out of town (one to Texas for 10 days), etc. The Texas trip was fun...besides all the great food I had a chance to setup computers for my new in-laws and it was refreshing to be a geek without the paycheck riding on it. It's fun coming up with analogies to explain why computers have such illogical behavior ("Why single-click here but double-click there? How come steering wheels are round? It just came that way.")
The last few weeks I've been digging into alternative languages and it's nice. Did you know Pascal is still alive and kicking out there? I've also been looking at Python, D, Ada, and Smalltalk. And in the office I had to do a bit of Perl. Then I read a bunch of "java sucks" rants online.
The net result is I feel so much freer thinking about code again. I'm curious now about how C++ templates work. I abandoned them in ShiftyNet for the sake of portability, but then again it would be nice to figure out how to get GCC to only create one implementation object file.
But I digress. Qodem is alive again. I finally got around to fixing the phonebook format to store attached notes to entries and cleared a few bugs. The next big item is upload support.
August 13, 2003
Gotta switch back to emacs... Renamed "dirty" to "q_screen_dirty", now all globals are prefixed with q_.
July 14, 2003
Meetings at work are wonderful opportunities for rote work on this project. I'm about 80% complete on gettext-izing console.c, after that it'll be phonebook.c and then I'll focus on fixing the known bugs.
June 25, 2003
Emacs is proving to be...interesting. Like the default behavior of untabify, NO! Gotta figure it out soon. I want END to be end-of-line, not end-of-file, likewise HOME. We'll get there eventually, in the meantime I got gettext working. Now, I'm going to force EVERYTHING through the gettext layer. This will mean the binary image of qodem won't work without the locale files (ooh, changes to the .deb) but it'll put English on equal footing with every other language.
June 24, 2003
Started gettext. Gonna begin switching to Emacs finally. (FYI my editor for the last four years has been 'joe'.)
June 19, 2003
Got the phonebook load function working. Added "RAW" capture type, about to add HTML. Gotta run to the auto insurance agent this afternoon though...
Released this code as 0.0.3. Bumping to 0.0.4. This next release is intended to close out some of the bugs and re-factor for a better long-term future.
June 7, 2003
Split screen is partially functional. Now adding directory list. Mostly functional now, just need to plug in the wildcard/regexp support.
June 3, 2003
Got the sort in, hideously inefficient but who's looking? It works, now I think it's split screen.
June 2, 2003
Started stubbing in the sort FON function. Got the call info working yesterday. I've gotta get to 0.0.4 ASAP, because I'm interviewing for positions at startups and as soon as I get an offer I'll be head down into the paid-for work until January.
May 28, 2003
I'm moving most of these kind of entries to the ChangeLog now. Added a debian package target.
May 27, 2003
Moved the carriage return and linefeed handling to emulation.c. That opens the door for a full vt100 later.
May 21, 2003
Stubbing in the screensaver. Hmm...got a small "issue": the screensaver won't know to kick in while blocking on keyboard input. Simple workaround: if the screensaver is on then the keyboard will never block. Not a big deal since the phone directory has most of the keyboard-intensive input. Except for the forms functions (notify_form(), save_form(), etc.). I'll open some bugs on SourceForge.
Got automake to create Makefile.in from Makefile.am. Not bad. Starting the debian package builder work.
May 20, 2003
Got the insert/delete working on the phonebook, along with pick lists for the method and emulation. Added an idle disconnect. Screen saver is the last feature before 0.0.2.
May 19, 2003
Got the phone entry revision screen in. Mostly works, except for choosing emulation and method, but fonebook.txt can be manually edited.
May 18, 2003
Stubbing in the phonebook screen now.
May 17, 2003
Added command-line support for the other connection methods, now we can actually be used as a telnet or ssh client.
May 16, 2003
Cleaned up the capture and screen dump routines to make use of open_workingdir_file(). Found some history regarding Qmodem(tm) from the BBS timeline on www.textfiles.com, put it in the README.
May 15, 2003
Moved the state-switching stuff to states.c.
Moved the ANSI emulation code to ansi.c. This will help us get VT52/VT100 later.
Alt-P Capture File now works.
May 14, 2003
Added a man page.
Alt-2 Backspace/Del now works. Alt-3 Line Wrap now works. Alt-4 Display NULL now works.
I was reading http://vt100.net and it looks like a "real" VT100 is quite difficult to do right. So I'm going to dedicate an entire point release to the VT100 emulation, maybe just going with the emulator from PuTTY or xterm. I'll save it for nearly-last since it'll take forever and a day to test.
May 13, 2003
Released 0.0.1. Feels nice, this is my first actual file release of any open-source project.
Alt-M Mail reader now works. Alt-- Status Lines now works.
May 12, 2003
Got the download screen working for Zmodem receive. Added keyboard emulation and beep support, so now bash shell works properly.
May 9, 2003
Stubbed in the download/upload window. Lunch, and then a quickie parser for rz's output. Current 'wc' output:
523 5417 30848 colors.c 120 595 4383 colors.h 20 98 579 config.h
1003 3199 28323 console.c 56 286 2067 console.h 194 641 4688 dialer.c 55 262 1800 dialer.h 787 2188 17782 emulation.c 84 392 2853 emulation.h 330 1206 9547 forms.c 46 251 1716 forms.h 53 264 1743 messages.c 60 303 2104 messages.h 416 1532 12364 options.c 63 263 1949 options.h 744 2359 20814 protocols.c 94 367 2835 protocols.h 610 1900 15252 qodem.c 130 513 3809 qodem.h 402 1330 10810 scrollback.c 73 367 2644 scrollback.h 88 427 2826 status.h 123 629 4128 tty.c 43 227 1514 tty.h 6117 25016 187378 total
Hmmm...getting a segfault on a large file. Doesn't seem to be any problems in the read()/write(), yet somehow memory is getting corrupted...hmm... oh well weekend time. I'm shooting for a 0.0.1 file release by this time next week.
May 8, 2003
Protocols are close but no cigar yet. There, got it. rz expects blocking stdin/stdout/stderr. I can transfer files successfully. Now to get the success to finish out and commit... Yup, needed to catch SIGPIPE and handle write() returning EPIPE when rz is gone.
May 7, 2003
Alt-C Clear screen.
Stubbed in the download/upload protocol stuff. I think I've figured out how to do rz/sz but damnit if it isn't behaving correctly. When I spawn the child process it freezes the main process and keyboard handlers/menus and refreshers stop working. After the child exits things are OK. Hmmm...maybe wait4() is blocking? Yup. So how can I check if a child is still running without calling wait4()? According to comp.unix.programming FAQ kill(pid,0) is the way to go. But that'll be a tomorrow thing...as my girlfriend is demanding the rest of the night. G'night.
May 6, 2003
Put in a simple notify screen for the mode switches. Alt-H works now, via SIGHUP. Next biggie: dialing directory. Today's todo: Unplug laptop from the wall outlet Give my girlfriend some squeeze time :)
Got scrollback save and clear working. Sourceforge CVS is down....grrr...
Alt-U Scrollback toggle.
May 5, 2003
Screen dump works. And forms.h has a terribly important piece of missing documentation: form field data won't be visible until form_driver(save_form, REQ_VALIDATION) has been called. Grrr! But we've got a generic form that will be used for downloads and capture too so I can't complain too much.
Did some other low-hanging fruit: linefeed on cr, 8th-bit strip, half/full duplex, and some status line fixes. The list of deviations from Qmodem(tm) is growing: good.
May 3, 2003
Got the ANSI color portion working, now putting in the cursor repositioning. Ran into a bunch if crap with the curses COLOR_PAIR(0) meaning white-on-black rather than black-on-black...grr...but I think it works now.
More cleaning. WIDTH and HEIGHT are now globals. Also got ansi_position() fixed so mc won't cause a core dump. Woo. Stubbed in the options file functions.
May 2, 2003
Starting the ANSI emulation now. There, now it strips all the codes it can recognize. (Just a subset of the full function but we can easily add more later.) This weekend's TODO is to actually modify the program variables so color comes in again.
OS shell works. Doorway works...sort of.
May 1, 2003
Scrollback buffer render is working, along with the info screen. Yippee! Today's TODO is to figure out how to get proper 8-bit characters from the child tty all the way through curses, then we can start worrying about ANSI.
April 30, 2003
Only one hour into it and I've got bash showing me ls! Weehee! All that's left of core function is scrollback, which I was thinking about last night as I went to sleep. The render is pretty easy, but I'm probably going to have to re-implement the storage at least twice. For now I don't care, let's get something up. Today's TODO: Be able to scroll the screen up one line when we are supposed to.
Wow. Got it running, took a bunch of stupid one-off errors, but believe it or not rendering the entire screen each time (and using printw() for EVERY character!) is still much faster on the console than a "real" bash prompt! Heehee.
Put in a top-level call to terminal emulation. Soon we'll have basic ANSI support. :) Time to commit...code size: 2169.
April 29, 2003
Got CVS access. Screwed up the initial import of course, lots of crap in the attic. Grr, anyway...
Fixed up the program flow, got some screenshots. Today's todo: Get the child TTY working Put in the console help screen, with unimplemented features in gray.
I'm about to call Santronics, the new owners of Mustang Software, to see if we can copy the help text from Qmodem 4.6 test drive. :) Their web site claims the # is 305-248-3204...dialing... press 2 for tech support ... Well gosh. Looks like Santronics didn't acquire Qmodem, that stayed with Mustang who got bought out by another company that later went bankrupt. So Qmodem is officially in dead-product-land, not legal to rip off the help screens. I'll need written permission from whoever bought Mustang... Back to coding, let's get the help screen in.
Done with the help screen. Let's play with TTYs now. I think I see what Eterm is doing, pretty clever. Got it! Tonight when I get home I'll get the execlp() call in and start dumping the child's output to the console. Boo. Ya. :)
April 28, 2003
SourceForge accepted the project, now waiting for CVS to be alive and well. Until then I've got the ALT- and ^Z/^C working, put in the global state variables, and have the beginnings of the console state working. Tomorrow I'll see about spawning a /bin/sh and piping its output around, then we'll boot to DOS and get some screenshots of Qmodem to model the menus after.
Patrick is listed as a developer, I'm really hoping he'll be interested in creating the scripting support. I figure a Lex/Yacc kind of thing would be in order, assuming it doesn't lead to huge code bloat. But I've never enjoyed compilers and that seems more his thing, so I'll defer design decisions if he does in fact come fully on board.
April 27, 2003
Project started. Registered on SourceForge, awaiting approval. GPL.
Got autoconf working mostly, need to add the check for ncurses. Initial window comes up, checks for keystrokes, exits on 'q'.
I don't think this will be an insanely difficult project. I suspect it will be useful for work within a week or two. Once I have the todos done below it'll just be adding the pluggable telnet/ssh/etc. and then a bunch of UI work.
Immediate todos: GNU gettext support (I18n) Detect alt- and ctrl- Signal handlers to detect ^Z and ^C TTY handler Allocate TTY Fork child /bin/sh Capture output