New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows,Skylark: write rule migration guide #3889

Open
laszlocsomor opened this Issue Oct 11, 2017 · 8 comments

Comments

@laszlocsomor
Contributor

laszlocsomor commented Oct 11, 2017

Write a guide on how to write portable Skylark rules.

@laszlocsomor

This comment has been minimized.

Show comment
Hide comment
@laszlocsomor

laszlocsomor Dec 14, 2017

Contributor

Bumping to P1 -- we committed to doing this work in Q4'2017.

Contributor

laszlocsomor commented Dec 14, 2017

Bumping to P1 -- we committed to doing this work in Q4'2017.

@laszlocsomor laszlocsomor added P1 and removed P2 labels Dec 14, 2017

@laszlocsomor

This comment has been minimized.

Show comment
Hide comment
@laszlocsomor

laszlocsomor Dec 14, 2017

Contributor

I plan to work on this next week.

Contributor

laszlocsomor commented Dec 14, 2017

I plan to work on this next week.

@j3parker

This comment has been minimized.

Show comment
Hide comment
@j3parker

j3parker Dec 14, 2017

Contributor

Some things I've hit:

  • Windows command line length limits. Many(?) tools use the "response file" idiom which (usually?) maps to Bazel's param files (via ctx.Args) directly (example PR: https://github.com/bazelbuild/rules_dotnet/pull/47/files )
  • ctx.actions.run should be used when possible (vs. the deprecated ctx.action) to avoid bash
  • Bash/etc. on windows should be avoided if possible (general perf, difficulty with consistent environment, weird edge cases like #4013)
  • Path length limits: a lot of tools (e.g. the C# compiler IIRC) are still very limited. I don't think this has come up yet for me in rule-writing specifically but maybe it could be a factor (we do set --output_user_root to C:\tmp to generally mitigate this.)

Something that would be interesting is any advice for making the best of unsandboxed execution. For example, we're able to vendor the C# compiler via an external package, but we rely on the system-wide .NET framework installation. My assumption (we haven't got to the point of cache sharing yet) is that these unsandboxed/unvendored resources won't contribute to the digests used in caching. This could lead (naturally) to non-repeatable builds in a way that could be dangerous with cache-sharing.

My guess at what I'll have to do is find some proxy to include as an input from outside the sandbox (e.g. there are files/registry keys that indicate the specific .NET framework release number) and "hope for the best". Any kind of insight here would be really helpful. If there's specific things we should document as rule authors on this topic that'd be good to know.

Contributor

j3parker commented Dec 14, 2017

Some things I've hit:

  • Windows command line length limits. Many(?) tools use the "response file" idiom which (usually?) maps to Bazel's param files (via ctx.Args) directly (example PR: https://github.com/bazelbuild/rules_dotnet/pull/47/files )
  • ctx.actions.run should be used when possible (vs. the deprecated ctx.action) to avoid bash
  • Bash/etc. on windows should be avoided if possible (general perf, difficulty with consistent environment, weird edge cases like #4013)
  • Path length limits: a lot of tools (e.g. the C# compiler IIRC) are still very limited. I don't think this has come up yet for me in rule-writing specifically but maybe it could be a factor (we do set --output_user_root to C:\tmp to generally mitigate this.)

Something that would be interesting is any advice for making the best of unsandboxed execution. For example, we're able to vendor the C# compiler via an external package, but we rely on the system-wide .NET framework installation. My assumption (we haven't got to the point of cache sharing yet) is that these unsandboxed/unvendored resources won't contribute to the digests used in caching. This could lead (naturally) to non-repeatable builds in a way that could be dangerous with cache-sharing.

My guess at what I'll have to do is find some proxy to include as an input from outside the sandbox (e.g. there are files/registry keys that indicate the specific .NET framework release number) and "hope for the best". Any kind of insight here would be really helpful. If there's specific things we should document as rule authors on this topic that'd be good to know.

@alexeagle

This comment has been minimized.

Show comment
Hide comment
@alexeagle

alexeagle Dec 14, 2017

alexeagle commented Dec 14, 2017

@laszlocsomor

This comment has been minimized.

Show comment
Hide comment
@laszlocsomor

laszlocsomor Dec 18, 2017

Contributor

Thank you @j3parker and @alexeagle , these are insightful and valuable ideas and questions. I'll incorporate them / their answers in the doc.

@j3parker :

Something that would be interesting is any advice for making the best of unsandboxed execution. For example, we're able to vendor the C# compiler via an external package, but we rely on the system-wide .NET framework installation. My assumption (we haven't got to the point of cache sharing yet) is that these unsandboxed/unvendored resources won't contribute to the digests used in caching. This could lead (naturally) to non-repeatable builds in a way that could be dangerous with cache-sharing.

Frankly I have no good answer at the moment. (For the record, we don't know about the feasibility of sandboxing on Windows.)

My guess at what I'll have to do is find some proxy to include as an input from outside the sandbox (e.g. there are files/registry keys that indicate the specific .NET framework release number) and "hope for the best".

That'd be one option. Another is to create a local repository for the .NET framework and write the .NET rules such that their actions depend on that local repository.

@alexeagle :

Thanks for mentioning runfiles, I'll write a section on that. There's also #4279.

Contributor

laszlocsomor commented Dec 18, 2017

Thank you @j3parker and @alexeagle , these are insightful and valuable ideas and questions. I'll incorporate them / their answers in the doc.

@j3parker :

Something that would be interesting is any advice for making the best of unsandboxed execution. For example, we're able to vendor the C# compiler via an external package, but we rely on the system-wide .NET framework installation. My assumption (we haven't got to the point of cache sharing yet) is that these unsandboxed/unvendored resources won't contribute to the digests used in caching. This could lead (naturally) to non-repeatable builds in a way that could be dangerous with cache-sharing.

Frankly I have no good answer at the moment. (For the record, we don't know about the feasibility of sandboxing on Windows.)

My guess at what I'll have to do is find some proxy to include as an input from outside the sandbox (e.g. there are files/registry keys that indicate the specific .NET framework release number) and "hope for the best".

That'd be one option. Another is to create a local repository for the .NET framework and write the .NET rules such that their actions depend on that local repository.

@alexeagle :

Thanks for mentioning runfiles, I'll write a section on that. There's also #4279.

@laszlocsomor

This comment has been minimized.

Show comment
Hide comment
@laszlocsomor

laszlocsomor May 16, 2018

Contributor

Update: sorry for having dropped the ball on this issue. I never submitted any docs. @j3parker and @alexeagle , do you think it's still worthwhile to publish best practices for writing platform-aware Skylark rules?

Contributor

laszlocsomor commented May 16, 2018

Update: sorry for having dropped the ball on this issue. I never submitted any docs. @j3parker and @alexeagle , do you think it's still worthwhile to publish best practices for writing platform-aware Skylark rules?

@alexeagle

This comment has been minimized.

Show comment
Hide comment
@alexeagle

alexeagle May 17, 2018

alexeagle commented May 17, 2018

@laszlocsomor

This comment has been minimized.

Show comment
Hide comment
@laszlocsomor

laszlocsomor May 18, 2018

Contributor

Thanks for the feedback. I'll try to get something done before the end of this quarter.

Contributor

laszlocsomor commented May 18, 2018

Thanks for the feedback. I'll try to get something done before the end of this quarter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment