lib/modules: better error message if an attr-set of options is expected

[Ma27]

Motivation for this change

I recently wrote some Nix code where I wrongly set a value to an option
which wasn't an actual option, but an attr-set of options. The mistake I
made can be demonstrated with an expression like this:

{
  foo = { lib, pkgs, config, ... }: with lib; {
    options.foo.bar.baz = mkOption {
      type = types.str;
    };
    config.foo.bar = 23;
  };
}

While it wasn't too hard to find the cause of the mistake for me, it was
necessary to have some practice in reading stack traces from the module
system since the eval-error I got was not very helpful:

error: --- TypeError --------------------------------------------------------- nix-build
at: (323:25) in file: /nix/store/3nm31brdz95pj8gch5gms6xwqh0xx55c-source/lib/modules.nix

   322|         foldl' (acc: module:
   323|                 acc // (mapAttrs (n: v:
      |                         ^
   324|                                    (acc.${n} or []) ++ f module v

value is an integer while a set was expected
(use '--show-trace' to show detailed location information)

I figured that such an error can be fairly confusing for someone who's
new to NixOS, so I decided to catch this case in th byName function in
lib/modules.nix by checking if the value to map through is an actual
attr-set. If not, a different error will be thrown.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[Ma27]

Ok, the tests only work with nixStable, sorry for that :)
I fixed the broken testcase now and also applied Sandro's suggestion.

[Ma27]

cc @infinisil in case you have the time rn, would you mind taking a look? :)

[infinisil]

Looking good, thanks!