atbash_cipher test suite is impossible(?) to satisfy

[macintux]

Unless I'm missing something, the requirements and the test are contradictory.

Specifically, you can't encode and then decode a sentence and expect the decoded sentence to match the original, since whitespace is not encoded, unless decoding involves guessing word boundaries, which is outside the scope.

I took a look at the Java tests to see how it evaluates this problem, and it provides expected encodings, not a full round trip.

Looking at the Nitpicks interface for Erlang, there are no submissions for this problem.

[macintux]

Ah, I can't see nitpicks for problems I haven't submitted, but I'm still skeptical this is a solvable problem.

[kytrinyx]

I'm pretty sure that decoding ignores word boundaries and just uses groups of five.

[macintux]

The test explicitly encodes and decodes the sentence, expecting the output to match the input. The specification is reasonable, but the test is not.

https://github.com/exercism/xerlang/blob/master/atbash-cipher/atbash_cipher_tests.erl#L16-L18 https://github.com/exercism/xerlang/blob/master/atbash-cipher/atbash_cipher_tests.erl#L16-L18

-John

On Aug 26, 2015, at 9:27 PM, Katrina Owen notifications@github.com wrote:

I'm pretty sure that decoding ignores word boundaries and just uses groups of five.

—
Reply to this email directly or view it on GitHub #62 (comment).

[kytrinyx]

Oh, yepp. I see what you mean. This was not included in the original exercise. I would suggest that we remove it.

[macintux]

As I poke around the other tests, there's very little consistency on how to treat underspecified cases.

E.g., the Ruby test removed whitespace between integers, while the Erlang test doesn't even expect punctuation to be removed. The Ruby test expects capitals to be lower-cased when encoded, while the Erlang test expects them to remain upper-case.

I'm preparing a PR for this test to make it a little less slanted towards cases the developer can't possibly expect to solve without looking at the test in advance, but I'd suggest this problem in general get a much clearer specification or be dropped entirely.

[macintux]

Aha, I figured it out.

The tests ignore this part of the specification:

Ciphertext is written out in groups of fixed length, the traditional group size
being 5 letters, and punctuation is excluded.

(Update: specifically, I mean the Erlang tests.)

[macintux]

Anyway, I don't know the protocol here. Would completely dropping these tests (and tweaking the README, since it makes impossible statements like "Decoding gsvjf rxpyi ldmul cqfnk hlevi gsvoz abwlt gives The quick brown fox jumps over the lazy dog.") in favor of tests ported from another language be a problem?

It's unclear to me how Exercism manages "epochs" in problems; if one user solves them using one test suite, and another using a different test suite, I can see potential confusion when nitpicking. On the other hand, since the tests are checked out just once, a user shouldn't care all that much if things change later.

And I don't see people downloading others' code to run against the test, so I guess my concern is largely unjustified.

Thoughts?

[kytrinyx]

Would completely dropping these tests (and tweaking the README [...]) in favor of tests ported from another language be a problem?

No, not at all. We're having a related discussion about clarity, purpose, focus, and consistency here: https://github.com/exercism/xjavascript/issues/105

It's unclear to me how Exercism manages "epochs" in problems

It's not been handled consistently. In most cases we've just sort of shrugged and decided that the point is to have an interesting and useful discussion, and that the code that is there is the code that is there. Unfortunately, we haven't communicated that clearly, which sometimes leads to people leading with "this doesn't pass the tests!", which is counter to the purpose of the site.

In a few tracks we've started introducing a simple versioning/book-keeping system, where we have a final test that asserts on some constant, and then the solution implements that constant. E.g.:

https://github.com/exercism/xruby/blob/master/hamming/hamming_test.rb#L81-L84

This at least gives nitpickers an indication that the version might be different from what they saw.

And I don't see people downloading others' code to run against the test, so I guess my concern is largely unjustified.

I do that sometimes. There's a hack for it in the CLI, too, because I was pretty fed up with copy/pasting:

exercism download SUBMISSION_ID

[NobbZ]

Since #66 is merged, can't this get closed?

[kytrinyx]

agreed, thanks.