make SDF to USD a separate component of sdformat #817

ahcorde · 2022-01-07T11:40:14Z

Signed-off-by: Ashton Larkin 42042756+adlarkin@users.noreply.github.com

🎉 New feature

Summary

Based on this PR #816 I made this other one smaller

The idea here it's to try to simplify things and be able to merge small PRs

This PR adds

USD as a module
CMD called sdf2usd to convert from SDF to USD
USD metadata

Checklist

Signed all commits for DCO
Added tests
Added example and/or tutorial
Updated documentation (as needed)
Updated migration guide (as needed)
codecheck passed (See contributing)
All tests passed (See test coverage)
While waiting for a review on your PR, please help review another open pull request to support the maintainers

Note to maintainers: Remember to use Squash-Merge and edit the commit message to match the pull request summary while retaining Signed-off-by messages.

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

Signed-off-by: ahcorde <ahcorde@gmail.com>

ahcorde · 2022-01-07T12:00:34Z

The module is not going to compile, the USD libraries are only available in source form from Pixar. NVIDIA is offering pre-built libraries (Linux and Windows) as well as the accompanying tools (e.g., USDView) for download.

I believe we should create our own binaries, isn't it @scpeters ? @j-rivero?

codecov-commenter · 2022-01-07T12:23:20Z

Codecov Report

Merging #817 (594b915) into sdf12 (0a15990) will not change coverage.
The diff coverage is n/a.

❗ Current head 594b915 differs from pull request most recent head 0851949. Consider uploading reports for the commit 0851949 to get more accurate results

@@           Coverage Diff           @@
##            sdf12     #817   +/-   ##
=======================================
  Coverage   90.70%   90.70%           
=======================================
  Files          78       78           
  Lines       12438    12438           
=======================================
  Hits        11282    11282           
  Misses       1156     1156

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0a15990...0851949. Read the comment docs.

j-rivero · 2022-01-07T16:01:46Z

I believe we should create our own binaries, isn't it @scpeters ? @j-rivero?

Interesting. I see different goals with respect to the installation and/or distribution of the USD pixar libraries:

We probably want to get them in CI as soon as possible. That could be achieved by using the pre-built binaries in our CI environments as the fastest method, if they are compatible. You can launch a quick test inside github actions probably.
To build, use and distribute our own binaries we need check the license. Seems a modified version of Apache-2 (I can not find what exactly was modified) but let's review carefully point 4. We can build Debian/Ubuntu/Brew packages, seems like at least vcpkg is already packaging them. The software comes from a a good bunch of embedded software on it, not sure if it exposed embedded symbols to the public API/ABI but we need to be careful with this.

adlarkin · 2022-01-07T17:21:02Z

The module is not going to compile, the USD libraries are only available in source form from Pixar. NVIDIA is offering pre-built libraries (Linux and Windows) as well as the accompanying tools (e.g., USDView) for download.

For further context, I tried to use NVIDIA's pre-built libraries/tools, and I had problems setting them up. I had to end up building USD from source. @j-rivero if you want to see how this was done in case it's useful for integrating this component with CI, I made a Dockerfile that sets up a USD source build: https://github.com/adlarkin/usd_docker

adlarkin

The overall structure looks good to me. I left some minor comments.

I am still seeing the following error when I run the component's installed sdf2usd executable:

sdf2usd: error while loading shared libraries: libusd_usd.so: cannot open shared object file: No such file or directory

usd/src/World.cc

examples/usdConverter/README.md

usd/src/cmd/CMakeLists.txt

usd/include/sdf/usd/CMakeLists.txt

usd/src/cmd/sdf2usd.cc

scpeters · 2022-01-07T23:00:46Z

I believe we should create our own binaries, isn't it @scpeters ? @j-rivero?

Interesting. I see different goals with respect to the installation and/or distribution of the USD pixar libraries:

We probably want to get them in CI as soon as possible.

I think we need to use the Pixar libraries in our CI before we can merge any of these pull requests. I can take a look at making a homebrew formula

koonpeng · 2022-01-10T08:00:34Z

For further context, I tried to use NVIDIA's pre-built libraries/tools, and I had problems setting them up. I had to end up building USD from source. @j-rivero if you want to see how this was done in case it's useful for integrating this component with CI, I made a Dockerfile that sets up a USD source build: https://github.com/adlarkin/usd_docker

nvidia's binaries may not work because it's compiled with pre cxx11 abi and a bunch of bundled dependencies (python3.6, boost1.70) which are not suitable for distribution (also, it includes gui programs like usdview which we don't need).

traversaro · 2022-01-10T08:27:40Z

I believe we should create our own binaries, isn't it @scpeters ? @j-rivero?

Interesting. I see different goals with respect to the installation and/or distribution of the USD pixar libraries:

* We probably want to get them in CI as soon as possible. That could be achieved by using the pre-built binaries in our CI environments as the fastest method, if they are compatible. You can launch a quick test inside github actions probably.

* To build, use and distribute our own binaries we need check the license. Seems a modified version of Apache-2 (I can not find what exactly was modified) but [let's review carefully point 4](https://github.com/PixarAnimationStudios/USD/blob/release/LICENSE.txt#L89-L128). We can build Debian/Ubuntu/Brew packages, seems like at least [vcpkg](https://vcpkg.info/port/usd) is already packaging them.

FYI I did a quick test to add usd to conda-forge some time ago, even if I did not worked a lot on it: conda-forge/staged-recipes#16666 .

The software comes from a a good bunch of embedded software on it, not sure if it exposed embedded symbols to the public API/ABI but we need to be careful with this.

Just to cross-link, there is an issue on usd issue tracker on that: PixarAnimationStudios/OpenUSD#1490 .

ahcorde · 2022-01-10T17:49:56Z

I was testing in this branch to compile USD from sources and at least check that the code is compiling. If you look for sdf2usd we will see how the module is compiling.

It's only compiling on focal because pxr requires the cmake version 3.12 and I was only able to install 3.10 on Bionic.

I can include this changes here but is increasing a lot the CI time

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

adlarkin · 2022-01-11T00:39:25Z

@ahcorde I finished up a "bare bones implementation" of usd::ParseSdfWorld in 18f3ac1, and added a unit test for verifying converted world information in 3bbd6ea. I believe that this creates a good baseline of functionality for this PR so that it can be merged into sdf12.

Regarding the tests, we still need to figure out how to address #796 (review). I'll see if I can figure out how to address this, and create a new PR if I come up with any ideas.

Signed-off-by: ahcorde <ahcorde@gmail.com>

ahcorde · 2022-01-11T15:04:55Z

@scpeters or @azeey Do you mind to review this PR ?

j-rivero · 2022-01-11T17:36:54Z

For further context, I tried to use NVIDIA's pre-built libraries/tools, and I had problems setting them up. I had to end up building USD from source. @j-rivero if you want to see how this was done in case it's useful for integrating this component with CI, I made a Dockerfile that sets up a USD source build: https://github.com/adlarkin/usd_docker

How much does it take to run that from-source build in a github action?

ahcorde · 2022-01-11T17:38:56Z

For further context, I tried to use NVIDIA's pre-built libraries/tools, and I had problems setting them up. I had to end up building USD from source. @j-rivero if you want to see how this was done in case it's useful for integrating this component with CI, I made a Dockerfile that sets up a USD source build: https://github.com/adlarkin/usd_docker

How much does it take to run that from-source build in a github action?

This is a build example https://github.com/ignitionrobotics/sdformat/runs/4776721844?check_suite_focus=true ~ 1hour, 7min

adlarkin

LGTM, but I do have a few final questions/comments that should be addressed before merging.

adlarkin · 2022-01-20T17:28:35Z

CMakeLists.txt

+
+  ########################################
+  # Find PXR
+  ign_find_package(pxr QUIET REQUIRED_BY usd PKGCONFIG pxr)


Configuration warnings should be gone now thanks to gazebo-tooling/release-tools#625, but I realize that until this PR is merged, gazebo-tooling/release-tools#625 will cause CI warnings on other sdformat PRs, as mentioned in gazebo-tooling/release-tools#625 (comment). I'm not sure if we should merge this PR first to take care of CI warnings on other PRs, or revert gazebo-tooling/release-tools#625 until it's time to merge this PR. Does anyone have thoughts on how to handle this?

usd/src/sdf2usd_TEST.cc

Signed-off-by: ahcorde <ahcorde@gmail.com>

azeey

LGTM with green CI. Looks like there's a test failure in Github Actions jobs.

usd/include/sdf/usd/World.hh

usd/src/World.cc

usd/include/sdf/usd/World.hh

azeey · 2022-01-22T00:15:05Z

usd/src/sdf2usd_TEST.cc

+    std::string output =
+      custom_exec_str(sdf2usdCommand() + " " + path + " " +
+        ignition::common::joinPaths(tmp, "shapes.usd"));


Has this been resolved @adlarkin ?

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

adlarkin

LGTM with green CI. Looks like there's a test failure in Github Actions jobs.

CI is green now - see #817 (comment)

The only other question I have before approving this is if we want to change the current file structure. This PR adds the SDF -> USD converter file structure, but this component will also have a USD -> SDF converter. So, I am wondering if we want sub-header and sub-source directories to differentiate between the SDF -> USD converter files and the USD -> SDF converter files: see #827 (review) for more information about this, @ahcorde and @azeey

.github/ci/before_cmake.sh

usd/include/sdf/usd/World.hh

Signed-off-by: ahcorde <ahcorde@gmail.com>

…nrobotics/sdformat into ahcorde/sdf_to_usd_cmake

Signed-off-by: ahcorde <ahcorde@gmail.com>

usd/include/sdf/usd/sdf_parser/World.hh

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

adlarkin

We should be good to merge if CI comes back green. Thanks for iterating, @ahcorde and @azeey!

osrf-triage · 2022-04-13T21:30:30Z

This pull request has been mentioned on Gazebo Community. There might be relevant details there:

https://community.gazebosim.org/t/new-ignition-releases-2022-04-13-fortress-edifice/1367/1

make SDF to USD a separate component of sdformat

5e13b49

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

ahcorde requested a review from azeey as a code owner January 7, 2022 11:40

ahcorde self-assigned this Jan 7, 2022

ahcorde requested a review from scpeters as a code owner January 7, 2022 11:40

ahcorde assigned adlarkin and koonpeng Jan 7, 2022

osrf-triage added this to Inbox in Core development Jan 7, 2022

github-actions bot added the 🏯 fortress Ignition Fortress label Jan 7, 2022

ahcorde mentioned this pull request Jan 7, 2022

Make SDF -> USD code a separate component of SDF #816

Closed

8 tasks

Added ignition-common4 as a dependency

fa4568b

Signed-off-by: ahcorde <ahcorde@gmail.com>

adlarkin reviewed Jan 7, 2022

View reviewed changes

chapulina moved this from Inbox to In review in Core development Jan 7, 2022

adlarkin mentioned this pull request Jan 8, 2022

Add sdf -> usd light functionality to usd component #818

Merged

8 tasks

adlarkin added 2 commits January 10, 2022 15:11

create world prim and add physics information

18f3ac1

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

add unit test for world conversion

3bbd6ea

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

ahcorde force-pushed the ahcorde/sdf_to_usd_cmake branch from 535a822 to 96caa08 Compare January 11, 2022 13:29

Added ci to compile with USD

16e0789

Signed-off-by: ahcorde <ahcorde@gmail.com>

ahcorde force-pushed the ahcorde/sdf_to_usd_cmake branch from 96caa08 to 16e0789 Compare January 11, 2022 14:33

azeey mentioned this pull request Jan 20, 2022

Skip pxr for sdformat gazebo-tooling/release-tools#625

Merged

ahcorde requested a review from adlarkin January 20, 2022 08:34

adlarkin reviewed Jan 20, 2022

View reviewed changes

Improved test

0195564

Signed-off-by: ahcorde <ahcorde@gmail.com>

ahcorde requested a review from adlarkin January 21, 2022 22:21

azeey reviewed Jan 22, 2022

View reviewed changes

adlarkin added 2 commits January 21, 2022 21:14

fix search path of sdf2usd executable in UNIT_sdf2usd_TEST

d91557e

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

alphabetize packages.apt

f3922f0

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

adlarkin reviewed Jan 22, 2022

View reviewed changes

koonpeng reviewed Jan 24, 2022

View reviewed changes

.github/ci/before_cmake.sh Outdated Show resolved Hide resolved

.github/ci/before_cmake.sh Outdated Show resolved Hide resolved

usd/include/sdf/usd/World.hh Outdated Show resolved Hide resolved

ahcorde added 5 commits January 24, 2022 09:01

cleanup before_cmake.sh

4c454a3

Signed-off-by: ahcorde <ahcorde@gmail.com>

changed file structure

95c07e9

Signed-off-by: ahcorde <ahcorde@gmail.com>

Merge branch 'ahcorde/sdf_to_usd_cmake' of https://github.com/ignitio…

b29b737

…nrobotics/sdformat into ahcorde/sdf_to_usd_cmake

File structure

c4c189f

Signed-off-by: ahcorde <ahcorde@gmail.com>

Added TODO

0851949

Signed-off-by: ahcorde <ahcorde@gmail.com>

adlarkin reviewed Jan 24, 2022

View reviewed changes

usd/include/sdf/usd/sdf_parser/World.hh Outdated Show resolved Hide resolved

adlarkin mentioned this pull request Jan 24, 2022

USD to SDF: Initial commit #827

Merged

8 tasks

fix include guard

01aeb57

Signed-off-by: Ashton Larkin <42042756+adlarkin@users.noreply.github.com>

adlarkin approved these changes Jan 24, 2022

View reviewed changes

azeey approved these changes Jan 24, 2022

View reviewed changes

adlarkin merged commit c80d78d into sdf12 Jan 24, 2022

Core development automation moved this from In review to Done Jan 24, 2022

adlarkin deleted the ahcorde/sdf_to_usd_cmake branch January 24, 2022 21:29

adlarkin mentioned this pull request Jan 26, 2022

convert SDF to USD #762

Closed

8 tasks

scpeters moved this from Done to Highlights in Core development Jan 31, 2022

adlarkin mentioned this pull request Feb 8, 2022

SDF -> USD conversion support #845

Closed

adlarkin mentioned this pull request Mar 23, 2022

USD to SDF: Fixed header guard #906

Merged

7 tasks

traversaro mentioned this pull request Mar 30, 2022

libsdformat v12.4.0 conda-forge/libsdformat-feedstock#56

Merged

3 tasks

j-rivero removed this from Highlights in Core development May 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make SDF to USD a separate component of sdformat #817

make SDF to USD a separate component of sdformat #817

ahcorde commented Jan 7, 2022 •

edited by adlarkin

Loading

ahcorde commented Jan 7, 2022

codecov-commenter commented Jan 7, 2022 •

edited

Loading

j-rivero commented Jan 7, 2022

adlarkin commented Jan 7, 2022 •

edited

Loading

adlarkin left a comment

scpeters commented Jan 7, 2022

koonpeng commented Jan 10, 2022

traversaro commented Jan 10, 2022 •

edited

Loading

ahcorde commented Jan 10, 2022

adlarkin commented Jan 11, 2022

ahcorde commented Jan 11, 2022

j-rivero commented Jan 11, 2022

ahcorde commented Jan 11, 2022

adlarkin left a comment

adlarkin Jan 20, 2022 •

edited

Loading

azeey left a comment

azeey Jan 22, 2022

adlarkin left a comment

adlarkin left a comment

osrf-triage commented Apr 13, 2022

make SDF to USD a separate component of sdformat #817

make SDF to USD a separate component of sdformat #817

Conversation

ahcorde commented Jan 7, 2022 • edited by adlarkin Loading

🎉 New feature

Summary

Checklist

ahcorde commented Jan 7, 2022

codecov-commenter commented Jan 7, 2022 • edited Loading

Codecov Report

j-rivero commented Jan 7, 2022

adlarkin commented Jan 7, 2022 • edited Loading

adlarkin left a comment

Choose a reason for hiding this comment

scpeters commented Jan 7, 2022

koonpeng commented Jan 10, 2022

traversaro commented Jan 10, 2022 • edited Loading

ahcorde commented Jan 10, 2022

adlarkin commented Jan 11, 2022

ahcorde commented Jan 11, 2022

j-rivero commented Jan 11, 2022

ahcorde commented Jan 11, 2022

adlarkin left a comment

Choose a reason for hiding this comment

adlarkin Jan 20, 2022 • edited Loading

Choose a reason for hiding this comment

azeey left a comment

Choose a reason for hiding this comment

azeey Jan 22, 2022

Choose a reason for hiding this comment

adlarkin left a comment

Choose a reason for hiding this comment

adlarkin left a comment

Choose a reason for hiding this comment

osrf-triage commented Apr 13, 2022

ahcorde commented Jan 7, 2022 •

edited by adlarkin

Loading

codecov-commenter commented Jan 7, 2022 •

edited

Loading

adlarkin commented Jan 7, 2022 •

edited

Loading

traversaro commented Jan 10, 2022 •

edited

Loading

adlarkin Jan 20, 2022 •

edited

Loading