Hyperparameters for Reproducing Evaluation Results

[StevenLau6]

Hi @luyang-huang96, thanks so much for posting the code.
Table 3 and 4 in your paper shows that the encoder variants (SINKHORN and LSH) can bring great performance.

1. To reproduce these results, I wonder if you use the hybrid attention in the encoder (how you set the input parameter encoder_not_hybrid)

2. If you use the hybrid attention in the encode (encoder_not_hybrid is false), I hope to know how you set the args.sw, args.encoder_linear, and args.encoder_kernel_linear.
   If only use the SINKHORN for all encoder layers (encoder_not_hybrid is True), my result shows its performance can not compete with LED/Bigbird, when using the same-length inputs.

LongDocSum/Model/longbart/longbartmodel.py

Lines 204 to 217 in d2b9bd0

	elif args.sinkhorn:
	if args.encoder_not_hybrid:
	self.layers.extend(
	[self.build_sinkhorn_layer(
	args, self.padding_idx)
	for i in
	range(args.encoder_layers)]
	)
	else:
	self.layers.extend(
	[self.build_sparse_encoder_layer(args, self._window[i], self.padding_idx) if i % 2 == 0 else self.build_sinkhorn_layer(args, self.padding_idx)
	for i in
	range(args.encoder_layers)]
	)