Add --skip-tag=... to have execution skip some tags

[akx]

Passing --skip-tag (can be passed multiple times) will have the executor entirely skip cells endowed with that tag.

That status is recorded in the result notebook (metadata.papermill.status = "skipped"), as well as the tags that contributed to this skipping (metadata.papermill.skip_tags).

cc @JuhaKiili

[casperdcl]

see #403

[casperdcl]

also note #402 (comment)

Generally it was considered a little too complex of an operation to manage in the open source space

I presume some @nteract devs have closed-source projects which implement such features and if the open source version catches up they won't be able to sell the closed-source version :)

[MSeal]

@casperdcl That's a bit harsh and not true in anyway. It's also insulting to the many people that spend lots of their free time working on these projects for no pay or specific recognition.

The point is that we want papermill to stay as simple as possible to avoid having a complex tool. This was a topic we decided to not incorporate directing into papermill. If you want to strip out cells before execution we suggest you use https://nbconvert.readthedocs.io/en/latest/removing_cells.html?highlight=RegexRemovePreprocessor#removing-cells-using-regular-expressions-on-cell-content to strip what you don't want before execution. This is an open source solution for this purpose, with nbconvert being the a much more complicated interface of options for transformation of notebooks. Additionally you can write an extension or adjacent open source solution -- we've helped many other groups in directing how to extend papermill for specific internal use-cases (free of any charge...).

[willingc]

@casperdcl Please respect the community values of the nteract project.

Like many popular projects in the Python science ecosystem, nteract is a NumFocus sponsored project. As part of nteract, papermill is a volunteer developed open source project.

I presume some @nteract devs have closed-source projects which implement such features and if the open source version catches up they won't be able to sell the closed-source version :)

Comments such as this one are unacceptable in our community. While you may have written this as sarcasm, it's difficult for a reader to determine if that is the case. Please refrain from making similar comments in the future. Thank you.

Brett Cannon has an excellent blog post and conference talk video on "Setting Expectations for Open Source Participation". There are many good points in his post.

[casperdcl]

Thanks @MSeal and @willingc for your clarification that closed source development is not the reason why these issues/PRs are not being incorporated. I am however surprised by the emotions this seems to have invoked.

That's a bit harsh

I don't think so; my statement was expressly "a guess" logically based on a direct comment.

and not true in anyway

Thank you for the clarification, as I said. All good up to here.

It's also insulting to the many people that spend lots of their free time working on these projects for no pay or specific recognition.

Erm. I don't think I'm going to debate this in the interest of keeping discussion here on-topic. I should however note that @akx, @aayush-jain18, and of course myself clearly spend lots of our free time working on projects for no pay or specific recognition. In our case we also feel ignored. This is not something I expect you to fix - making people feel included in this case is not a legal requirement - and it is up to you how you want to take your project forward.

Comments such as this one are unacceptable in our community.

I contest this completely. Comments such as these should be expressly encouraged to start open discussion and clarification. I also strongly contest the implication that anything that I have said or implied has gone against the community values (or indeed the Code of Conduct), both of which I have read in detail.

While you may have written this as sarcasm, it's difficult for a reader to determine if that is the case. Please refrain from making similar comments in the future.

I was not being remotely sarcastic - purely earnest. I cannot in good faith refrain from making politely earnest remarks, but will do my best as always to ensure they are not misinterpreted as meaning something else.

On the topic of misinterpretation, I take it that @MSeal has retracted the original comment:

Generally it was considered a little too complex of an operation to manage in the open source space

due to its misleading implication that closed-source development is ongoing?

[MSeal]

Generally it was considered a little too complex of an operation to manage in the open source space

is not retracted -- there's missing context in capturing just the quote that the solution of skipping cells within papermill's common execution engine was deemed a little to complex. Nbconvert's solution is to generate a notebook without those cells, then separately process it and is in a project that has more complexity than papermill. The open source space part was referring to this open source project with this approach to the problem, which to be fair was not stated explicitly in responses.

I felt your comments were not following:

• Gracefully accepting constructive criticism
• Being respectful of differing viewpoints and experiences
• Showing empathy towards other community members

You stated I had pushed back on an idea as one of the maintainers of this project because I wished to instead make people buy software that nteract devs produce without evidence to the fact beyond a restatement around not choosing to add this to our open source code. This implies a level of deviousness in my response and some intent to subvert the goals of the project that from my position did not feel respectful nor empathetic.

Alternatives you could take to accusing a member of having malicious intent:

• Ask a question about how one might approach the problem your facing if this project doesn't want to add the capability
• Ask if there are any other reasons this feature isn't being added
• See how one could make an add-on capability outside of papermill in open or closed source to enable the functionality (a lot of internal platforms at companies use this approach for specific capabilities they wish to have for their platform which may not be additive to the open source tool)

Please take @willingc and my statements as "we both felt that statement was not up to our standards of communication within the project". Regardless of if you felt it was appropriate, others did not and as a community it's as important 'how your message is received' as it is 'what you intended'. If you want to continue to conversation or have concrete objections I'd also ask you to take the conversation out of this PR's thread and communicate directly with the maintainers.

[MSeal]

Apologies to @akx for having this long discussion on the thread here. I am probably going to ask to close this PR, even though functionally it's perfectly sound for the project scope reasons linked above if that's alright.

[willingc]

Hi folks,

I have opened up an issue #429 to gather use cases about when skipping cell execution may be helpful. Documenting these use cases would be a reasonable next step toward understanding the motivation for this functionality as well as whether alternative approaches for skipping cells could be taken. I will leave issue #429 open for 30 days for public comments and then we can summarize the use cases and determine options either within papermill or another project.

I am closing this PR for now until the responses are captured in #429. We can reopen it for modification, if needed, after the discussion in #429. Thanks @akx and @MSeal. 🌻