New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding a test to show supported data types in TensorFlowTransform #2101

Merged
merged 6 commits into from Jan 16, 2019

Conversation

Projects
None yet
5 participants
@zeahmed
Copy link
Member

zeahmed commented Jan 10, 2019

This PR adds addition test to fix #745.

@zeahmed zeahmed requested review from Ivanidzo4ka , abgoswam , yaeldekel and sfilipi Jan 10, 2019

var mlContext = new MLContext(seed: 1, conc: 1);
// Pipeline
var loader = ComponentCreation.CreateDataView(mlContext,
new List<TestDataInt>(new TestDataInt[] { new TestDataInt() { a = new[] { 1L, 2,

This comment has been minimized.

@wschin

wschin Jan 10, 2019

Member

Not aligned? #Resolved

Assert.Equal(4 + 4, cValues[3]);

Assert.True(cursor.MoveNext());
c = default;

This comment has been minimized.

@yaeldekel

yaeldekel Jan 10, 2019

Member

c = default; [](start = 16, length = 12)

This line is not needed. #Resolved

VBuffer<long> c = default;
cgetter(ref c);

var cValues = c.GetValues();

This comment has been minimized.

@yaeldekel

yaeldekel Jan 10, 2019

Member

cValues [](start = 20, length = 7)

Add an assert for the length of this. #Resolved

c = default;
cgetter(ref c);

cValues = c.GetValues();

This comment has been minimized.

@yaeldekel

yaeldekel Jan 10, 2019

Member

cValues [](start = 16, length = 7)

Add an assert for the length of this. #Resolved

/// Test to ensure the support for int and int64 types.
/// </summary>
[ConditionalFact(typeof(Environment), nameof(Environment.Is64BitProcess))] // TensorFlow is 64-bit only
public void TensorFlowTransformAdditionOfIntAndInt64Test()

This comment has been minimized.

@singlis

singlis Jan 11, 2019

Member

TensorFlowTransformAdditionOfIntAndInt64Test [](start = 20, length = 44)

Hi Zeeshan, is there supposed to be a test for the 32-bit model? I thought the other tests were 32-bit models, but It looks like the other tests either are skipped or disable unless its a 64-bit process.
#Resolved

This comment has been minimized.

@zeahmed

zeahmed Jan 11, 2019

Member

TensorFlow works only on 64-bit platform. So, i think none of the TF tests are 32-bit.


In reply to: 246982838 [](ancestors = 246982838)

This comment has been minimized.

@singlis

singlis Jan 11, 2019

Member

ah! sorry I misread, you are handling int64 and int32 types...so then do the other tests use models that are using int32s?


In reply to: 246982838 [](ancestors = 246982838)

This comment has been minimized.

@singlis

singlis Jan 11, 2019

Member

actually I see -- you have a TestDataInt class that has both long and int....


In reply to: 246983378 [](ancestors = 246983378,246982838)

@singlis
Copy link
Member

singlis left a comment

:shipit:

@Ivanidzo4ka

This comment has been minimized.

Copy link
Member

Ivanidzo4ka commented Jan 11, 2019

Tests are good, but original issue is not about tests, and more about, what types we can actually support?
Can we support bool? Can we support short?
Do we have list of supported types in our documentation?
Do we have any exceptions in case if user passes type which we don't support? Not the cryptic one, but one which would say: "Current type "A" is unsupported please use following types: float, int, int64"? #Resolved

@zeahmed

This comment has been minimized.

Copy link
Member

zeahmed commented Jan 11, 2019

documentation is here:

<description>Currently, float, double, int, long, uint, ulong are the acceptable data types for input/output.</description>

I believe short is also supported. I will add it to the documentation.
Its tedious to write tests for all the types so I created a tests for int/int64 to test for the most common types. But I will see if I can create a generic model that can used to test all the supported types.
I will verify if exceptions are properly thrown. I think if there is a mismatch between the type and type is not supported an exception is thrown.


In reply to: 453388399 [](ancestors = 453388399)

@wschin

This comment has been minimized.

Copy link
Member

wschin commented Jan 11, 2019

Can Tensorflow transform work with key types? They are integers as well. #Resolved

@zeahmed zeahmed changed the title Adding a test to show TensorFlowTransform can work with int32 and int64 types Adding a test to show supported data types in TensorFlowTransform Jan 14, 2019

@zeahmed

This comment has been minimized.

Copy link
Member

zeahmed commented Jan 14, 2019

Updated the test to show how the supported types can be used.


In reply to: 453611413 [](ancestors = 453611413,453388399)

@zeahmed

This comment has been minimized.

Copy link
Member

zeahmed commented Jan 14, 2019

No, if they can be represented as any of the these types (double, float, long, int, short, sbyte, ulong, uint, ushort, byte and bool) then yes.


In reply to: 453671331 [](ancestors = 453671331)

@Ivanidzo4ka
Copy link
Member

Ivanidzo4ka left a comment

:shipit:

@zeahmed zeahmed merged commit 68841ad into dotnet:master Jan 16, 2019

2 checks passed

MachineLearning-CI #20190115.31 succeeded
Details
license/cla All CLA requirements met.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment