Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix Issue 10378 - Local imports hide local symbols #5445

Merged
merged 1 commit into from
Feb 16, 2016

Conversation

WalterBright
Copy link
Member

This is a fix of https://issues.dlang.org/show_bug.cgi?id=10378 and a reboot of #4915

It requires dlang/phobos#3989

This version is based on a comment Andrei made to me a long time ago about it. Symbol lookup is now done in two passes (formerly one). The first pass goes up through the scopes, checking everything but imported modules. If that fails, the scopes are gone through again, checking only imported modules.

This algorithm:

  1. retains the scoped nature of nested imports, i.e. names in nested imports will hide outer imported names
  2. works the same for function locals as it does for struct members, which was my complaint about Issue 10378 - Local imports hide local symbols #4915
  3. does not issue errors for shadowing - it simply finds the shadowed symbol first.
  4. breaks existing code. Turns out that people did rely on it. See fix problems that prevent implementing import shadowing protection phobos#3989
  5. is pretty simple. (the large number of lines of code changed is mostly due to debugging and printing code which I'll remove once it passes the autotester
  6. exposed a bug in the way the compiler is implemented, see fix problems that prevent implementing import shadowing protection phobos#3989 for that, too. I'm sure there'll be other instances in the wild of this.
  7. it should only be a hair slower, as it never looks in the same hash table twice

@dlang-bot
Copy link
Contributor

Fix Bugzilla Description
10378 Prevent local imports from hiding local symbols

@WalterBright
Copy link
Member Author

Looks like @deadalnix deadalnix proposed this algorithm here:

https://issues.dlang.org/show_bug.cgi?id=10378#c9

{
Dsymbol sx = searchScopes(flags); // look in both locals and imports
version (LOGSEARCH)
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove version none and move initialising sx into version LOGSEARCH?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that code will be removed when it passes the autotester

@deadalnix
Copy link
Contributor

Yes, that's how it needs to be !

@dnadlinger
Copy link
Member

We should make fixing this and issues 313/314 the main point of this release.

@WalterBright
Copy link
Member Author

Needs this Phobos fix:
dlang/phobos#3990

Folks, this language change is clearly going to break existing code.

@andralex
Copy link
Member

nice work!

@andralex
Copy link
Member

Auto-merge toggled on

@schveiguy
Copy link
Member

When this is merged, I think we need a doc change too.

@9rnsr
Copy link
Contributor

9rnsr commented Feb 12, 2016

Auto-merge toggled off

@9rnsr
Copy link
Contributor

9rnsr commented Feb 12, 2016

I want to review this.

@@ -0,0 +1,4 @@

module bar;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A module bar; is imported by import imports.bar10378;. It's a mismatch.

I'm not sure why the error is not reported, but at least, this line should be: module imports.bar10378;.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's actually a feature of D. The file name doesn't have to match the module name.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know the feature, but this is not its case. This file is imported as the module name imports.bar10378, but the module declaration has a different fully qualified name bar which does not match.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's actually a feature of D. The file name doesn't have to match the module name.

It's actually the result of a horribly broken implementation. The module is inserted w/ both names into the global symbol table.

I'm not sure why the error is not reported

The error is only reported if the module is know under the module declaration name before importing it, this happens when it's parsed before the module importing it.

@9rnsr
Copy link
Contributor

9rnsr commented Feb 12, 2016

@WalterBright : Also I give you a chance to remove debugging code from this PR.

@Geod24
Copy link
Member

Geod24 commented Feb 12, 2016

@WalterBright : Is there any chance we get a deprecation for the old behaviour ?

I agree that is the right direction to go, and that old code relied on a bug. However, that code was not necessarily broken / buggy, which is why I don't think breaking it without a proper deprecation is okay.

@@ -1,3 +0,0 @@
module imports.diag12598a;

struct lines { }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's better to move to compilable directory, rather than deleting test case for issue 12598.

Of course adding a comment about that in commit log is even better.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It doesn't test for anything novel anymore and serves no purpose. Hence, deleting it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should record into the git history what happened in this PR. Moving the test case to compiable would be the best clarify way that the issue 12598 case is now just accepted.
If you don't like it, you should add a comment into the commit message, like "issue 12598 case is now properly accepted, so its test case fail_compilation/diag12598.d is deleted.

@9rnsr
Copy link
Contributor

9rnsr commented Feb 12, 2016

The most of PR summary needs to be copied into commit log, no?

@9rnsr
Copy link
Contributor

9rnsr commented Feb 12, 2016

Excepting my comments, the implementation itself looks good to me.

@WalterBright
Copy link
Member Author

add another switch

Please, no:

  1. complexity
  2. documentation
  3. it is not as easy as you suggest - note that every override of search() changed the flags.
  4. if a user sees new "undeclared symbol" messages, he can simply add the -transition=import switch and if the error goes away, he knows that's what the problem is.

BTW, doing a fork has more problems. This change touches a lot of files, and I've had to rebase it twice in 4 days just so it will merge. I'm not keen on the time investment in this.

@schveiguy
Copy link
Member

I don't have much to offer here, except support for this change. As much as this will break code, people have been clamoring for import issues to be fixed for almost a decade. I think folks will understand this is going to help them even though their code breaks.

Regarding build issues, I think those are somewhat secondary to the ability to have code that compiles for both older and newer versions of the compiler. Simply because the two alternative compilers lag behind DMD. So as long as there is a way to fix broken code so it compiles for the new scheme and the old scheme, I'd say passing the right flags is a secondary concern.

@MartinNowak
Copy link
Member

Your arguments are correct, the very few issues this caused ¹ relied on rather obscure behavior, and the new lookup rules are a huge improvement.
The code is also clearer now, and we have a nice explanation in the changelog.

@MartinNowak
Copy link
Member

Auto-merge toggled on

@JackStouffer
Copy link
Member

Auto-merge toggled on

Thanks Walter for doing this!

@schveiguy
Copy link
Member

I have a question on this. If I want to have an inner scope where I add an overload via an import, how would I do it? Currently, you have to re-import the original so they are on the same level. From my reading of the new rules, this would continue to be the case.

An example of what I'm talking about is in the bug report this PR is fixing (at the end of the comment): https://issues.dlang.org/show_bug.cgi?id=10378#c16

MartinNowak added a commit that referenced this pull request Feb 16, 2016
fix Issue 10378 - Local imports hide local symbols
@MartinNowak MartinNowak merged commit 57592cf into dlang:master Feb 16, 2016
@WalterBright
Copy link
Member Author

Currently, you have to re-import the original so they are on the same level.

That's right (and this PR does not change that behavior).

Edit: Functions that exist in two imports do not actually overload against each other. They form separate "overload sets". The anti-hijacking rules keep them distinct. The only way you'd want to overload against both is if the scope wants to actually call both, in which case it makes sense to require re-importing just for clarity.

@WalterBright
Copy link
Member Author

So as long as there is a way to fix broken code so it compiles for the new scheme and the old scheme,

I had no problems doing that for the Phobos library breakage.

@schveiguy
Copy link
Member

Right. The only confusing part is to require importing the current module you are in (because another module hijacked a module symbol you were using).

Interestingly, this doesn't happen when you import at module level. The current module trumps any imports. But when you import at a scope, then the rules change. As a user of library code, I can see value in importing a module locally when it only pertains to that particular function. What I'm concerned with (and am still, even though I like the PR as merged), is that a movement of an import from the module level to a scoped level changes what happens.

example:

module a;

void foo() {import std.stdio; writeln("a");}

void bar() {}
module b;
import a;

void foo() {import std.stdio; writeln("b");}

void main()
{
    bar();
    foo();
}

This outputs "b"

Hey, look, we can just import a locally in main, because we only need bar in there!

module b;

void foo() {import std.stdio; writeln("b");}

void main()
{
    import a;
    bar();
    foo();
}

This now outputs "a".

I'm unsure of the "fix" for this, but it may be good practice to recommend always importing within an inner scope using renamed imports (to avoid hijacking like this).

@WalterBright WalterBright deleted the fix10378-2 branch February 17, 2016 03:13
@WalterBright
Copy link
Member Author

@schveiguy Your second example now outputs b with this PR. This fix is the whole point of this PR.

@schveiguy
Copy link
Member

our second example now outputs b with this PR

Oh right! I wasn't thinking about the fact that local module functions aren't actually imported :)

So no need to reimport the current module if the current module's function is what you want. This is great! Thanks.

@CyberShadow
Copy link
Member

Could someone please tell me if this breakage is intentional or should I file a regression?

int memcmp(in ubyte[] a, in ubyte[] b)
{
    import core.stdc.string;
    assert(a.length == b.length);
    return memcmp(a.ptr, b.ptr, a.length);
}

This no longer works because it tries to call itself - not what I would expect TBH, seems like a bug to me.

@schveiguy
Copy link
Member

I think it's intentional, as imports now are considered after local module functions.

I disagree that it's a bug. If you imported core.stdc.string in the module, the imported memcmp wouldn't be considered.

@CyberShadow
Copy link
Member

If you imported core.stdc.string in the module, the imported memcmp wouldn't be considered.

That would not be surprising. The behavior in the above snippet is surprising, though, because the import is "closer" (in scope terms) to the use of the memcmp symbol than the function declaration.

Changing the import to a selective import makes it compile again, which is weird, as I always thought that selective imports are simply selective versions of imports, and do not grant additional visibility to the selected symbols.

I kind of see why this change was necessary, though, but the additional complexity and code breakage isn't nice.

@schveiguy
Copy link
Member

Yes, selective import actually aliases the symbol (and only the one you selected) into your local scope, which would override any other imported symbols and local symbols.

It is going to be very confusing, and code-breaky, because it's been this way for so long. I don't think there's a way we could do this without breaking code.

@schveiguy
Copy link
Member

Note: I always looked at it the opposite way -- that importing in a local scope should not change how the import was done, just what scopes could see the import. Moving an import inside a function shouldn't change the behavior of the function, you are just trying to avoid polluting namespaces (you can see my example above).

@ibuclaw
Copy link
Member

ibuclaw commented Mar 8, 2016

@schveiguy are you sure? core.stdc.string.memcmp has a different signature to the function that imports it's module. I can see no reason why that snippet would even consider recursively calling itself.

@WalterBright
Copy link
Member Author

Could someone please tell me if this breakage is intentional or should I file a regression?

It's behaving exactly as intended.

has a different signature

Signatures are considered for overload resolution, not for name lookup. D has always worked this way, and so has C++.

@MartinNowak
Copy link
Member

Turn on -transition=checkimports, it should give you a proper deprecation warning about the changed lookup.

@CyberShadow
Copy link
Member

Turn on -transition=checkimports, it should give you a proper deprecation warning about the changed lookup.

I am seeing new deprecation warnings even without this flag (using dmd master). Is the flag meant for stable and the breaking changes for master, or something else?

@CyberShadow
Copy link
Member

It's behaving exactly as intended.

OK. Another question: does a selective import not imply a static one? E.g.:

import std.stdio : readln;
import std.conv;

void main()
{
    std.stdio.writeln("Hello, world!");
}

This now gives:

test.d(6): Deprecation: module std.stdio is not accessible here, perhaps add 'static import std.stdio;'

but that strikes me as redundant considering the selective import.

@schveiguy
Copy link
Member

does a selective import not imply a static one?

No it was not supposed to. But this is due to a different pull:

#5426

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet