New lint `clippy::allow_attributes`

[Jefffrey]

Is your feature request related to a problem or challenge?

Lint reference: https://rust-lang.github.io/rust-clippy/master/index.html?search=allow_att#allow_attributes

Checks for usage of the #[allow] attribute and suggests replacing it with the #[expect] attribute (See RFC 2383)

#[expect] attributes suppress the lint emission, but emit a warning, if the expectation is unfulfilled. This can be useful to be notified when the lint is no longer triggered.

We have a quite a few occurrences of #[allow] in our codebase, but should prefer #[expect] instead for the reasons above.

Describe the solution you'd like

Similar to #18503 we should roll this lint our gradually through crates before finally enabling at the workspace level so we don't have one big PR.

Describe alternatives you've considered

Don't enable this lint if we don't see value or if the false positive rate is too high (see below).

Additional context

I tested it for some of our current #[allow]s and it has some false positives, especially for dead_code and deprecated.

Example 1

datafusion/datafusion/core/benches/data_utils/mod.rs

Lines 37 to 40 in c8d26ba

	/// create an in-memory table given the partition len, array len, and batch size,
	/// and the result table will be of array_len in total, and then partitioned, and batched.
	#[allow(dead_code)]
	pub fn create_table_provider(

• If I switch this to #[expect(dead_code)] then clippy will flag it as this lint expectation is unfulfilled. But if I remove the lint then the dead code warning pops up anyway 🤔

Example 2

datafusion/datafusion/execution/src/disk_manager.rs

Lines 117 to 136 in c8d26ba

	/// Configuration for temporary disk access
	#[allow(deprecated)]
	#[deprecated(since = "48.0.0", note = "Use DiskManagerBuilder instead")]
	#[derive(Debug, Clone, Default)]
	pub enum DiskManagerConfig {
	/// Use the provided [DiskManager] instance
	Existing(Arc<DiskManager>),
	/// Create a new [DiskManager] that creates temporary files within
	/// a temporary directory chosen by the OS
	#[default]
	NewOs,
	/// Create a new [DiskManager] that creates temporary files within
	/// the specified directories
	NewSpecified(Vec<PathBuf>),
	/// Disable disk manager, attempts to create temporary files will error
	Disabled,
	}

• Switching this to #[expect(deprecated)] causes similar to above, but also flags NewOS variant as using deprecated variant. Something to do with how Default is derived?

(I say false positive but it seems more like a bug in clippy 🤔)

Maybe there are solutions for the above, I haven't looked into it much yet.

Another thing to keep in mind is how features interact, for example here:

datafusion/datafusion/datasource-parquet/src/reader.rs

Lines 290 to 304 in c8d26ba

	fn get_metadata<'a>(
	&'a mut self,
	#[allow(unused_variables)] options: Option<&'a ArrowReaderOptions>,
	) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {
	let object_meta = self.partitioned_file.object_meta.clone();
	let metadata_cache = Arc::clone(&self.metadata_cache);
	async move {
	#[cfg(feature = "parquet_encryption")]
	let file_decryption_properties = options
	.and_then(|o| o.file_decryption_properties())
	.map(Arc::clone);
	#[cfg(not(feature = "parquet_encryption"))]
	let file_decryption_properties = None;

• This #[allow] is for when parquet_encryption feature is not enabled, but if we change it to #[expect] blindly then I believe it'll cause issues when parquet_encryption is enabled, so we need to be careful

Tasks

• datafusion
• datafusion-catalog
• datafusion-catalog-listing
• datafusion-common
• datafusion-common-runtime
• datafusion-datasource
• datafusion-datasource-arrow
• datafusion-datasource-avro
• datafusion-datasource-csv
• datafusion-datasource-json
• datafusion-datasource-parquet
• datafusion-doc
• datafusion-execution
• datafusion-expr
• datafusion-expr-common
• datafusion-ffi
• datafusion-functions
• datafusion-functions-aggregate
• datafusion-functions-aggregate-common
• datafusion-functions-nested
• datafusion-functions-table
• datafusion-functions-window
• datafusion-functions-window-common
• datafusion-macros
• datafusion-optimizer
• datafusion-physical-expr
• datafusion-physical-expr-adapter
• datafusion-physical-expr-common
• datafusion-physical-optimizer
• datafusion-physical-plan
• datafusion-proto
• datafusion-proto-common
• datafusion-pruning
• datafusion-session
• datafusion-spark
• datafusion-sql
• datafusion-substrait
• Enable it globally after all the above tasks are finished

Note: we would welcome PRs that can do multiple crates in a single PR (so long as the diff isn't too large). Felt a bit overwhelmed by the volume and speed of PRs that came in for #18503 😅

[Jefffrey]

Thoughts @2010YOUY01 ?

[2010YOUY01]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

[Jefffrey]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

It would look something like this:

    fn get_metadata<'a>(
        &'a mut self,
        #[cfg_attr(not(feature = "parquet_encryption"), expect(unused_variables))]
        options: Option<&'a ArrowReaderOptions>,
    ) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {

Doable, but a little verbose.

[2010YOUY01]

If there is a good way to deal with the mentioned feature flag issue, I think we should add this lint rule.

It would look something like this:

fn get_metadata<'a>(
    &'a mut self,
    #[cfg_attr(not(feature = "parquet_encryption"), expect(unused_variables))]
    options: Option<&'a ArrowReaderOptions>,
) -> BoxFuture<'a, parquet::errors::Result<Arc<ParquetMetaData>>> {

Doable, but a little verbose.

I think this marker describes the arg quite precisely: 'this arg is only used in parquet_encryption' feature.

So I'm +1 for this lint. Maybe we should add comment like the above explanation, to deal with the verbosity.

[Jefffrey]

I've marked it as good first issue, but note to anyone interested, please do not assign this issue to yourself as it is meant to be a tracking issue. Just pick a crate (or crates) that is not yet done/in-progress and leave a comment with your intention to work on those crates (to avoid overlap).

[YuraLitvinov]

Hello, if it's okay I am taking the physical-plan.