Ensure File.open applies default umask on gem extract

[martinemde]

This is an alternate to #7299

What was the end-user or developer problem that led to this PR?

Global and group write permissions are possible in gems.

What is your fix for the problem, implemented in this PR?

Allow File.open to handle applying the default umask.

Non-backwards-compatible change: This drops previously tested persistence of extended file modes.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[deivid-rodriguez]

This looks good to me!

The only thing intentionally being reverted is 3211b16. I agree it's best to respect umask by default, and I would treat this as a security fix, but let me ask @nobu in case there was some reason we are missing for that change.

@nobu Is it ok for us to revert 3211b16? Do you remember of any issues with the default umask being respected? Perhaps it was just for consistency with it not being respected in other places?

[martinemde]

@deivid-rodriguez @nobu If there's a problem with this PR, #7299 would be the right one to switch to, with maybe an intentional mode & 0o700 added.

Is there a test we could add?

[deivid-rodriguez]

mmm so in few words, #7299 is more backwards compatible (defining that by our existing tests) but less secure?

[martinemde]

Yes. On my system #7299 doesn't clear setuid/setgid bits on files. Allowing File.open to do its default clears those bits. This implies that File.umask (usually 0o022 or 0o077) is a limited version (9 bits?) of the actual umask that is applied by File.open (which is something more like 0o7022 but is probably much more platform dependent)

[deivid-rodriguez]

Thanks for explaining. Are you comfortable then with shipping this as a security fix? I think I am.

[martinemde]

I prefer this version, but there's an untested implied behavior in the original commit by @nobu that may mean some gems on some systems don't get their 0700 set.

(Edit: I take that back, the chmod will set to prog_mode or 0755 no matter what. I'm not sure about the 0700...)

[deivid-rodriguez]

I see. Let's wait for @nobu's feedback then.

[hsbt]

This change break some of our CI https://app.travis-ci.com/github/ruby/ruby/jobs/615527743