Add .NET functional tests to Mint #64

poornas · 2017-07-01T02:35:54Z

Functional tests for minio-dotnet Sdk.

this is wip as multipart uploads are failing on the docker image but not on native environment.

harshavardhana · 2017-07-02T03:19:58Z

A basic question how are we thinking of making dotnet tests run with mint?

nitisht · 2017-07-02T03:30:44Z

So @poornas proposed that we have a separate Dockerfile that is based on windows image and just runs dotnet tests.

My understanding is that a separate Dockerfile is needed because of Mono (on Linux) doesn't currently work for .NetCore 4.5.2 that we support. @poornas can weigh in more about it.

nitisht · 2017-07-03T02:24:25Z

blocked till further discussion

nitisht · 2017-07-03T18:20:31Z

Not blocked anymore. As discussed, looks like there is a way to run .Net tests on mono and hence the current Docker image itself. @poornas will send the updated PR

nitisht · 2017-07-04T05:32:37Z

@poornas can you pls add WIP to the title until new changes are added.

Just to clarify there is work to be done here.

poornas · 2017-07-05T03:00:21Z

this PR can be reviewed - 2 wrinkles though.

The dotnet dependencies are not being cleaned up because the dotnet publish step doesnt generate a executable.
multipart put/copy are failing - this is an SDK problem and a separate issue has been raised. The 3-4 tests that fail have been commented out for now.

krisis

Overall changes look good. Disposing MemoryStream needs to be handled in some places.
Test result:
No executable found matching command "dotnet-/mint/run/core/minio-dotnet/out/Minio.Functional.Tests.dll"

krisis · 2017-07-05T11:13:49Z

run/core/minio-dotnet/FunctionalTest.cs

+                Console.WriteLine("[Bucket]  Exception: {0}", e);
+                Assert.Fail();
+            }
+            await TearDown(minio, bucketName);


Shouldn't filestream be disposed in the finally block or alternately use using block to implicitly dispose?

@krisis, done.

nitisht · 2017-07-05T18:10:49Z

run/core/minio-dotnet/FunctionalTest.cs

+        private static int MB = 1024 * 1024;
+
+        // Create a file of given size from random byte array
+        private static String CreateFile(int size)


Remove local CreateFile

Preferably use streaming as done here https://github.com/minio/minio-py/pull/547/files#diff-123f6df576c99ae259903f693e5bc6a4R34

For tests where Static files are needed, use /mint/data files

Another example would be this https://github.com/minio/minio-java/pull/586/files

poornas · 2017-07-06T19:49:12Z

@nitisht, updated PR to use streaming wherever possible.

poornas · 2017-07-08T01:11:41Z

@harshavardhana ,@nitisht - PR's #146,#149 and #150 on minio-dotnet repo need to be merged to master and released before I can remove the local copy of Functional Tests in the mint/run/core/minio-dotnet directory. There are also a couple of TODO's to uncomment before this PR is merged to Mint.Until then, this PR is blocked but can be reviewed.

nitisht · 2017-08-02T08:25:21Z

@poornas as minio/minio-dotnet#146, minio/minio-dotnet#149, minio/minio-dotnet#150 are merged. Marking this as unblocked.

If there is nothing to add here, we can review this.

poornas · 2017-08-12T23:53:22Z

@nitisht , need this PR minio/minio-dotnet#162 to merge as well into master before we can run dotnet tests on mint.

nitisht · 2017-08-29T02:15:13Z

@poornas minio/minio-dotnet#162 is merged. Can we move ahead on this

nitisht · 2017-09-12T02:17:05Z

@poornas I am just running the Docker image generated by travis

sudo docker run  -e "SKIP_TESTS=aws-sdk-php,mc,awscli,aws-sdk-ruby,minio-java,minio-js,minio-py,minio-go" -e "ENABLE_HTTPS=0" --net=host play.minio.io/mint:travis-6888a325d0000c83796b2d338f40cca9b10d63bb

Can you please try this and confirm if you're not getting the error.

poornas · 2017-09-13T05:15:32Z

@nitisht , yes I see the error with the command you posted - for play the tests.sh script sets the MINT_DATA_DIR variable triggering the symlink bug. I updated my other PR for additional functional tests on dotnet sdk to also eliminate use of symlinks.

nitisht · 2017-11-24T06:20:31Z

@poornas can you pls fix the conflicts? We can take this in as well before we release Mint.

poornas · 2017-11-25T04:33:37Z

@nitisht , conflicts were fixed.PTAL.

nitisht · 2017-11-25T04:44:21Z

build/minio-dotnet/install.sh

+set -e
+
+MINIO_DOTNET_SDK_PATH="$MINT_RUN_CORE_DIR/minio-dotnet"
+MINIO_DOTNET_SDK_VERSION="1.0.3"


We can probably follow the approach to automatically pull the latest SDK version, as done in other SDKs. Like this

MINIO_DOTNET_VERSION=$(curl -s https://api.github.com/repos/minio/minio-dotnet/releases/latest | jq -r .tag_name) if [ -z "$MINIO_DOTNET_VERSION" ]; then echo "unable to get minio-dotnet version from github" exit 1 fi

can we do this @poornas

nitisht · 2017-11-25T05:15:15Z

Tested locally run perfectly fine

SERVER_ENDPOINT: 127.0.0.1:9000
ACCESS_KEY:      minio
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp 170bbcf375e8:/mint/log /tmp/mint-logs'
(1/11) Running aws-sdk-go tests ... done in 0 seconds
(2/11) Running aws-sdk-php tests ... done in 41 seconds
(3/11) Running aws-sdk-ruby tests ... done in 1 seconds
(4/11) Running awscli tests ... done in 24 seconds
(5/11) Running mc tests ... done in 4 seconds
(6/11) Running minio-dotnet tests ... done in 40 seconds
(7/11) Running minio-go tests ... done in 29 seconds
(8/11) Running minio-java tests ... done in 6 seconds
(9/11) Running minio-js tests ... done in 21 seconds
(10/11) Running minio-py tests ... done in 10 seconds
(11/11) Running s3cmd tests ... done in 6 seconds

All tests ran successfully

However there is a stray line printed in json logs

Listed all delete errors for remove objects on miniodotnetv1hnnurml6a9rqe

We need to remove this for the log to be parseable

nitisht · 2017-11-25T08:38:57Z

Sent minio/minio-dotnet#192
Once that is merged, we can release minio-dotnet. This will fix the logging issues.

nitisht · 2017-11-25T08:40:04Z

build/minio-dotnet/install.sh

+MINIO_DOTNET_SDK_PATH="$MINT_RUN_CORE_DIR/minio-dotnet"
+MINIO_DOTNET_SDK_VERSION="1.0.3"
+
+outDir="$MINIO_DOTNET_SDK_PATH/out"


out_dir is more idiomatic (Ref: https://google.github.io/styleguide/shell.xml#Variable_Names)

ok, changed this.

nitisht · 2017-11-25T08:40:13Z

install-packages.list

+apt-transport-https
+dotnet-dev-1.0.4
+nuget
+libunwind8


new line missing

nitisht · 2017-11-25T08:40:29Z

remove-packages.list

@@ -4,3 +4,5 @@ python3-pip
 ruby-dev
 ruby-bundler
 default-jdk
+dotnet
+nuget


new line missing

nitisht

LGTM & Tested

harshavardhana · 2017-11-28T07:04:28Z

@poornas can you fix this issue?

Decompressing 100% 5567 ms
Expanding 100% 18013 ms
/mint/run/core/minio-dotnet/Minio.Functional.Tests.csproj(1,1): error MSB4025: The project file could not be loaded. Data at the root level is invalid. Line 1, position 1.
The command '/bin/sh -c /mint/build/minio-dotnet/install.sh' returned a non-zero code: 1

nitisht

Travis build failures need to be fixed..

nitisht · 2017-11-29T11:58:29Z

Built and tested locally, works fine

SERVER_ENDPOINT: 127.0.0.1:9000
ACCESS_KEY:      minio
SECRET_KEY:      ***REDACTED***
ENABLE_HTTPS:    0
SERVER_REGION:   us-east-1
MINT_DATA_DIR:   /mint/data
MINT_MODE:       core

To get logs, run 'sudo docker cp 2e0336d2f1cf:/mint/log /tmp/mint-logs'
(1/11) Running aws-sdk-go tests ... done in 0 seconds
(2/11) Running aws-sdk-php tests ... done in 41 seconds
(3/11) Running aws-sdk-ruby tests ... done in 2 seconds
(4/11) Running awscli tests ... done in 24 seconds
(5/11) Running mc tests ... done in 3 seconds
(6/11) Running minio-dotnet tests ... done in 39 seconds
(7/11) Running minio-go tests ... done in 24 seconds
(8/11) Running minio-java tests ... done in 6 seconds
(9/11) Running minio-js tests ... done in 20 seconds
(10/11) Running minio-py tests ... done in 13 seconds
(11/11) Running s3cmd tests ... done in 7 seconds

All tests ran successfully

nitisht requested review from krisis and harshavardhana July 2, 2017 02:52

nitisht added the blocked label Jul 3, 2017

nitisht removed the blocked label Jul 3, 2017

deekoder requested review from nitisht and removed request for krisis July 4, 2017 00:37

poornas changed the title ~~Add minio-dotnet functional tests~~ [wip]Add minio-dotnet functional tests Jul 4, 2017

poornas changed the title ~~[wip]Add minio-dotnet functional tests~~ [wip] Add minio-dotnet functional tests Jul 4, 2017

poornas force-pushed the mdotnet branch from 935dc79 to ca9aaa3 Compare July 5, 2017 02:43

poornas changed the title ~~[wip] Add minio-dotnet functional tests~~ Add minio-dotnet functional tests Jul 5, 2017

poornas force-pushed the mdotnet branch from ca9aaa3 to 45d76bb Compare July 5, 2017 02:48

krisis reviewed Jul 5, 2017

View reviewed changes

nitisht suggested changes Jul 5, 2017

View reviewed changes

poornas mentioned this pull request Jul 7, 2017

Evaluate if we can use windows base image as a docker container. #30

Closed

poornas changed the title ~~Add minio-dotnet functional tests~~ [wip] Add minio-dotnet functional tests Jul 8, 2017

poornas added the blocked label Jul 8, 2017

poornas changed the title ~~[wip] Add minio-dotnet functional tests~~ [wip] Add .NET functional tests to Mint Jul 19, 2017

poornas force-pushed the mdotnet branch from 548cad6 to 8624ccf Compare July 25, 2017 01:01

nitisht removed the blocked label Aug 2, 2017

poornas closed this Sep 13, 2017

poornas reopened this Sep 13, 2017

poornas force-pushed the mdotnet branch 2 times, most recently from 8012bbd to 2f5a932 Compare November 24, 2017 10:56

nitisht reviewed Nov 25, 2017

View reviewed changes

nitisht mentioned this pull request Nov 25, 2017

Fix json logging for Mint minio/minio-dotnet#192

Merged

nitisht reviewed Nov 25, 2017

View reviewed changes

poornas force-pushed the mdotnet branch from 2f5a932 to d2315af Compare November 27, 2017 01:29

poornas changed the title ~~[wip] Add .NET functional tests to Mint~~ Add .NET functional tests to Mint Nov 27, 2017

poornas force-pushed the mdotnet branch from d2315af to 54e518d Compare November 28, 2017 06:47

nitisht approved these changes Nov 28, 2017

View reviewed changes

poornas force-pushed the mdotnet branch 2 times, most recently from 57b116d to ca2c01b Compare November 28, 2017 07:26

nitisht suggested changes Nov 28, 2017

View reviewed changes

poornas force-pushed the mdotnet branch from ca2c01b to 00255bc Compare November 28, 2017 17:52

add dotnet functional tests

68454e3

poornas force-pushed the mdotnet branch from 00255bc to 68454e3 Compare November 29, 2017 07:10

nitisht approved these changes Nov 29, 2017

View reviewed changes

harshavardhana approved these changes Nov 29, 2017

View reviewed changes

nitisht merged commit 9bdc5cb into minio:master Nov 29, 2017

Add .NET functional tests to Mint #64

Add .NET functional tests to Mint #64

Conversation

poornas commented Jul 1, 2017

harshavardhana commented Jul 2, 2017

nitisht commented Jul 2, 2017

nitisht commented Jul 3, 2017

nitisht commented Jul 3, 2017

nitisht commented Jul 4, 2017

poornas commented Jul 5, 2017

krisis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitisht Jul 5, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

poornas commented Jul 6, 2017

poornas commented Jul 8, 2017

nitisht commented Aug 2, 2017 • edited Loading

poornas commented Aug 12, 2017

nitisht commented Aug 29, 2017

nitisht commented Sep 12, 2017

poornas commented Sep 13, 2017 • edited Loading

nitisht commented Nov 24, 2017

poornas commented Nov 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitisht commented Nov 25, 2017

nitisht commented Nov 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nitisht left a comment

Choose a reason for hiding this comment

harshavardhana commented Nov 28, 2017

nitisht left a comment

Choose a reason for hiding this comment

nitisht commented Nov 29, 2017

nitisht Jul 5, 2017 •

edited

Loading

nitisht commented Aug 2, 2017 •

edited

Loading

poornas commented Sep 13, 2017 •

edited

Loading