Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration tests #11276

Merged
merged 3 commits into from Aug 28, 2019

Conversation

@jamesbeyond
Copy link
Contributor

commented Aug 21, 2019

Description

During 2019 SiP workshop, we introduced a set of tests as integration testing on storage and network. We asked partner to run these tests on their platforms fix issues if any. Generally these tests received very positive feedback, and we decide to merge the test into mbed-os master

Pull request type

[ ] Fix
[ ] Refactor
[x] Target update
[ ] Functionality change
[ ] Docs update
[ ] Test update
[ ] Breaking change

Reviewers

@MarceloSalazar @maclobdell

@ciarmcom ciarmcom requested review from maclobdell, MarceloSalazar and ARMmbed/mbed-os-maintainers Aug 21, 2019
@ciarmcom

This comment has been minimized.

Copy link
Member

commented Aug 21, 2019

@jamesbeyond, thank you for your changes.
@maclobdell @MarceloSalazar @ARMmbed/mbed-os-maintainers please review.


## Execute tests

The integration test requires both the connectivity and and storage device to to configured.

This comment has been minimized.

Copy link
@MarceloSalazar

MarceloSalazar Aug 21, 2019

Contributor

minor - remove duplicate 'and' and 'to'

This comment has been minimized.

Copy link
@MarceloSalazar

MarceloSalazar Aug 21, 2019

Contributor

still pending to remove duplicate 'and'

Copy link
Contributor

left a comment

Overall looks good - I've suggested a few minor changes in the readme doc.

@MarceloSalazar

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

@iriark01 do you want to have a quick look to the readme doc?

@iriark01

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

@AnotherButler might have time. If this needs to go in the public docs, it will also need to move repos.

Copy link
Member

left a comment

Commits could provide more details. The first one is adding 14 files - what these are? At least an introduction for each new test unit in this commit would be nice to have.

License for alice text is not acceptable, we shall find a replacement for the file used.


This eBook is for the use of anyone anywhere at no cost and with
almost no restrictions whatsoever. You may copy it, give it away or
re-use it under the terms of the Project Gutenberg License included

This comment has been minimized.

Copy link
@0xc0170

0xc0170 Aug 21, 2019

Member

We had a discussion previously about this one as I recall. To my understanding, the license is not acceptable.

This comment has been minimized.

Copy link
@jamesbeyond

jamesbeyond Aug 21, 2019

Author Contributor

This file got removed already, it just exist in the git history. do you want me clean up the history and force push a commit as well ? @0xc0170

This comment has been minimized.

Copy link
@0xc0170

0xc0170 Aug 21, 2019

Member

I see, I was probably reviewing only the first commit after I checked what it includes. please remove from history . Will review afterwards

@jamesbeyond jamesbeyond force-pushed the sip-workshop branch from 797ebf0 to e2a77a6 Aug 21, 2019
@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

Hi @MarceloSalazar @MarceloSalazar @AnotherButler ,
your comments been addressed, please review again.

@MarceloSalazar

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

Reviewed - pending to remove duplicate 'and'.

@AnotherButler

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2019

We need to move the README file content to the mbed-os-5-docs repository. I can do this tomorrow or Friday.

Could someone please explain sample.txt? Why are we copying content from mbed-os-5-docs and pasting it here?

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

We need to move the README file content to the mbed-os-5-docs repository. I can do this tomorrow or Friday.

Could someone please explain sample.txt? Why are we copying content from mbed-os-5-docs and pasting it here?

Hi @AnotherButler , thanks for reviewing, the sample.txt is the guinea pig, the test case is trying to send this file over the network, then verify the revived copy same as the original one.

BTW, why do we need to move the README to docs repo? the README only related to the tests inside this folder. it might not fit in to the contents of mbed-os-5-docs ? what do you think @MarceloSalazar

@0xc0170

This comment has been minimized.

Copy link
Member

commented Aug 22, 2019

BTW, why do we need to move the README to docs repo? the README only related to the tests inside this folder. it might not fit in to the contents of mbed-os-5-docs ? what do you think @MarceloSalazar

I was going to ask similar question. My assumption would be to have intergration tests documented in docs and referenced in Readme.md ? However looking at tests, usb have extensive readme for instance similar to these integration tests.
@AnotherButler what shall we do. Clear guidance for tests would be appreciated.

@MarceloSalazar

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

I'd love the documentation to be available in os.mbed.com/docs, as it will raise awareness and instruct developers/partners on how to run the tests. At the moment, it's hidden/buried and it's not easy to find it.

Copy link
Member

left a comment

Why the directory is "COMMON" rather than "common"? Upper case has special meaning in our build system and it may be confusing.

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

Why the directory is "COMMON" rather than "common"? Upper case has special meaning in our build system and it may be confusing.

Yeah, I think that's the reason we keep it as COMMON, otherwise build system will treat this folder as a test, and trying to build it, but failed in the end

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

I'd love the documentation to be available in os.mbed.com/docs, as it will raise awareness and instruct developers/partners on how to run the tests. At the moment, it's hidden/buried and it's not easy to find it.

Currently it feels fragmentation in the testing contents in mbeds documentation.
maybe we can have a section in mbed-os-5-docs, focusing on the testing contents, explain to people what it which and how to run them ?

@AnotherButler

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

We do have a section in our published documentation about testing: https://os.mbed.com/docs/mbed-os/v5.13/tools/testing.html

It describes our different types of tests and how to build and write them.

If we publish this content there in our published documentation, then it should live in that repository. Having two copies of the same content leads to content out of sync.

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

HI @AnotherButler, I the one you pointed out is the test tooling, but not the actual tests itself.
For example, these tests I am trying to merge is about integration test on Storage and Connection.
and as far as I know, there are testing about

They are not in the mbed-os-5-docs

@MarceloSalazar

This comment has been minimized.

Copy link
Contributor

commented Aug 22, 2019

The integration tests are critical and involve many components - IMO deserves its own section in the docs in the porting section.
I'd suggest moving the content under the porting process here: https://os.mbed.com/docs/mbed-os/v5.13/porting/index.html

@0xc0170 0xc0170 added needs: work and removed needs: review labels Aug 23, 2019
@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

Hi @MarceloSalazar, do you suggest we move the docs from this repo to mbed-os-5-docs repo? or do we leave the docs here and duplicate the README in the docs repo ?

@MarceloSalazar

This comment has been minimized.

Copy link
Contributor

commented Aug 23, 2019

We shouldn't have any duplication.
mbed-os-5-docs is the best place for this document, IMO.

@jamesbeyond

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

I remove the README file, and moved it to docs repo. This PR should be ready for review and CI @MarceloSalazar @0xc0170

@0xc0170

This comment has been minimized.

Copy link
Member

commented Aug 26, 2019

CI started

@0xc0170 0xc0170 changed the title Merge integration tests into mbed-os master Add integration tests Aug 26, 2019
@mbed-ci

This comment has been minimized.

Copy link

commented Aug 26, 2019

Test run: FAILED

Summary: 1 of 4 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_greentea-test
@MarceloSalazar MarceloSalazar self-requested a review Aug 27, 2019
@0xc0170

This comment has been minimized.

Copy link
Member

commented Aug 27, 2019

Restarted tests

@0xc0170

This comment has been minimized.

Copy link
Member

commented Aug 27, 2019

CI restarted

@mbed-ci

This comment has been minimized.

Copy link

commented Aug 27, 2019

Test run: SUCCESS

Summary: 4 of 4 test jobs passed
Build number : 2
Build artifacts

@0xc0170 0xc0170 added ready for merge and removed needs: CI labels Aug 28, 2019
@0xc0170 0xc0170 merged commit d2c0591 into master Aug 28, 2019
18 checks passed
18 checks passed
continuous-integration/jenkins/pr-head This commit looks good
Details
jenkins-ci/build-ARM Success
Details
jenkins-ci/build-GCC_ARM Success
Details
jenkins-ci/build-IAR Success
Details
jenkins-ci/greentea-test Success
Details
travis-ci/astyle Success!
Details
travis-ci/docs Success!
Details
travis-ci/doxy-spellcheck Success!
Details
travis-ci/events Success! Runtime is 8336 cycles.
Details
travis-ci/gitattributestest Success!
Details
travis-ci/include_check Success!
Details
travis-ci/licence_check Success!
Details
travis-ci/littlefs Success! Code size is 8464B.
Details
travis-ci/psa-autogen Success!
Details
travis-ci/tools-py2.7 Success!
Details
travis-ci/tools-py3.5 Success!
Details
travis-ci/tools-py3.6 Success!
Details
travis-ci/tools-py3.7 Success!
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.