Implement project for Transform.

[liurenjie1024]

See java implementation.

What's project used for? Say we have row filter a = 10, and we have a partition spec bucket(a, 37) as bs, if one row matches a=10, then its partition value should match bucket(10, 37) as bs, and we project a=10 to bs = bucket(10, 37).

Blocked by #283

[marvinlanhenke]

@liurenjie1024
...just to make sure I'm on the right track - and to confirm the idea.

Here is what I would (high-level):

• implement fn project on Transform
• match Transform & orginal BoundPredicate
• handle each Transform (e.g. Bucket) and Predicate according to Java Implementation
• return Option<Predicate>

In order to implement the ProjectionEvaluator later, we can then get the PartitionSpec from the manifest, iterate over fields: Vec<PartitionField> and call fn project(...) on each field's transform...

some pseudo-implementation to illustrate:

impl Transform {
// ...
pub fn project(&self, name: String, pred: &BoundPredicate) -> Option<Predicate> {
        match self {
            Transform::Bucket(_) => match pred {
                BoundPredicate::Unary(expr) => Some(Predicate::Unary(UnaryExpression::new(
                    expr.op(),
                    Reference::new(name),
                ))),
                _ => unimplemented!(),
            },
            _ => unimplemented!(),
        }
    }
}

#[test]
fn test_bucket_project() {
    let trans = Transform::Bucket(8);

    let name = "projected_name".to_string();

    let field = NestedField::required(1, "a", Type::Primitive(PrimitiveType::Int));

    let pred = BoundPredicate::Unary(UnaryExpression::new(
        PredicateOperator::IsNull,
        BoundReference::new("original_name", Arc::new(field)),
    ));

    let result = trans.project(name, &pred).unwrap();

    assert_eq!(format!("{result}"), "projected_name IS NULL");
}

[liurenjie1024]

@marvinlanhenke Sorry for late reply. Yeah, this is exactly what I'm thinking about, thanks!

[marvinlanhenke]

thank you, once #283 is ready, I will continue here.

[marvinlanhenke]

@ZENOTME
are you still working on #287 and the proposed interface?
just looking for a quick update on this - and huge thanks for your effort on this.

[ZENOTME]

@ZENOTME are you still working on #287 and the proposed interface? just looking for a quick update on this - and huge thanks for your effort on this.

Yes. I will update it later.

[marvinlanhenke]

@liurenjie1024 @ZENOTME
...the fn project(...) is getting kinda extensive (lengthy). I think it'd be appropriate to introduce some helper functions here instead of writing a huge match statement - to make the code more readable and reduce duplication.

Any suggestions/ preferences where to put those?

• mod.rs in iceberg/src/transform
• create utils.rs in iceberg/src/spec
• create fns on impl Transform itself

thanks for your thoughts on this

EDIT:
For now, I decided to go with option no. 3, which I like since the scope of those helper functions is restricted only to Transform - putting those into some utils.rs would not be justified, since nobody else woud need those fns...

[liurenjie1024]

@liurenjie1024 @ZENOTME ...the fn project(...) is getting kinda extensive (lengthy). I think it'd be appropriate to introduce some helper functions here instead of writing a huge match statement - to make the code more readable and reduce duplication.

Any suggestions/ preferences where to put those?

• mod.rs in iceberg/src/transform
• create utils.rs in iceberg/src/spec
• create fns on impl Transform itself

thanks for your thoughts on this

EDIT: For now, I decided to go with option no. 3, which I like since the scope of those helper functions is restricted only to Transform - putting those into some utils.rs would not be justified, since nobody else woud need those fns...

Hi, @marvinlanhenke Thanks for raising this. I prefer option no. 3 too with similar reason: putting related codes together helps to organize things better.

[ZENOTME]

@liurenjie1024 @ZENOTME ...the fn project(...) is getting kinda extensive (lengthy). I think it'd be appropriate to introduce some helper functions here instead of writing a huge match statement - to make the code more readable and reduce duplication.

Any suggestions/ preferences where to put those?

• mod.rs in iceberg/src/transform
• create utils.rs in iceberg/src/spec
• create fns on impl Transform itself

thanks for your thoughts on this

EDIT: For now, I decided to go with option no. 3, which I like since the scope of those helper functions is restricted only to Transform - putting those into some utils.rs would not be justified, since nobody else woud need those fns...

+1 for no. 3 too