Cleanup VARIANT marshalling and convert to managed #102498
Conversation
dotnet-policy-service
bot
added
the
community-contribution
| private static unsafe object? ConvertWrappedObject(object? wrapped) => wrapped switch | ||
| { | ||
| _objref = null; | ||
| _flags = CV_U4; | ||
| _data = val; | ||
| } | ||
|
|
||
| public Variant(long val) | ||
| null or Empty => null, | ||
| Enum => wrapped.GetType(), | ||
| IntPtr or UIntPtr => wrapped, | ||
| ValueType v when RuntimeHelpers.GetMethodTable(v)->IsPrimitive => null, | ||
| DateTime or TimeSpan or Currency => null, | ||
| _ => wrapped, | ||
| }; |
There was a problem hiding this comment.
Explanation for this:
Previously, when UnknownWrapper or DispatchWrapper is passed in, the wrapped object goes through SetFieldObject, and the objRef field will be used as the real object passed to native.
Our managed test is exercising one case, but not exhaustive:
|
The failures looks unrelated now. |
| _flags = CV_I8; | ||
| _data = val; | ||
| } | ||
| // Cases handled at native side: string, bool, primitives including (U)IntPtr excluding U(Int)64, array |
There was a problem hiding this comment.
If this is the case, I would handle them in the switch and throw or assert so we know what is going on.
There was a problem hiding this comment.
The question is the caller from RECORD case of MarshalHelperCastVariant.
The pre-existing behavior is convert whatever received to VARIANT, and use VariantChangeType to attempt to convert it to VT_RECORD. However, VariantChangeType doesn't allow anything else to cast to VT_RECORD.
To determine whether an object will be casted to VT_RECORD, I'm making a call to MarshalHelperConvertObjectToVariant to execute all the rule in it. So these types are end up hittable and will be handled by the IConvertible case.
There was a problem hiding this comment.
I also have a question about observable exception throwing behavior. Getting unmarshallable types (TimeSpan, SafeHandle etc) passed into CastVariant will throw corresponding exception. I'm currently only passing value types because they won't create things that need clean-up in the VARIANT.
| // Helper code: on the back propagation path where a VT_BYREF VARIANT* | ||
| // is marshaled to a "ref Object", we use this helper to force the | ||
| // updated object back to the original type. | ||
| internal static void MarshalHelperCastVariant(object pValue, int vt, ref Variant v) | ||
| internal static void MarshalHelperCastVariant(object pValue, int vt, out ComVariant v) |
There was a problem hiding this comment.
Can we return here too?
Co-authored-by: Aaron Robinson <arobins@microsoft.com>
| case VarEnum.VT_RECORD: | ||
| MarshalHelperConvertObjectToVariant(pValue, out v); | ||
| if (v.VarType != VarEnum.VT_RECORD) | ||
| throw new InvalidCastException(SR.InvalidCast_CannotCoerceByRefVariant); | ||
| break; |
There was a problem hiding this comment.
I'm trying to restore the behavior here, and relying on the location to be cleaned up by RAII at native side.
Is there any better option for this?
There was a problem hiding this comment.
What is the previous behavior in this case? I'm not following the issue being hit here.
There was a problem hiding this comment.
I don't see any destructors in MarshalOleRefVariantForObject. Is that where you are talking about?
There was a problem hiding this comment.
Previously in MarshalOleRefVariantForObject, if SafeVariantChangeType fails, would the data (potentially BSTR or something else) in vtmp be leaked? SafeVariantClear won't be called in throwing path.
I've concluded that SafeVariantChangeType is unnecessary and will always fail. The content of the VARIANT should be cleared at managed side in failing path.
|
@AaronRobinsonMSFT Can you answer remaining questions in this PR? The concern is about cleanup in exceptional cases with partial result. When taking a ref at managed side, RAII at native side may take care of the cleanup. |
Cleans up the managed Variant struct used as an intermediate value.
Basically, folds
MarshalHelperConvertObjectToVariantwithMarshalOleVariantForComVariant,MarshalHelperConvertVariantToObjectandMarshalHelperCastVariantwithMarshalComVariantForOleVariant.Tests were first written and tested on main.
Array and record marshalling are still kept native. VT_ARRAY marshalling looks sharing many logic with non-variant marshalling.