RuntimeError: CUDA error: an illegal memory access was encountered Compile with TORCH_USE_CUDA_DSA to enable device-side assertions
#228
Comments
|
Did you make any changes to simple_stacking.py? |
|
@balakumar-s |
|
Does your robot work with other examples like |
balakumar-s
added
question
|
I am having the same problem. I also replaced the Franka Panda with a different robot but even if I put the Panda back in it still does not work. Interestingly, the bug is not consistent. It sometimes happens, and sometimes doesn't. |
|
Have you tried with the latest commit? We pushed a CUDA kernel fix that might resolve this issue. |
|
It seems to be resolved for now. Could be due to having restarted the docker container or could be due to the CUDA kernel fix - I will see. Either way, thank you! |
|
The problem came back. Could you please reopen the issue? |
|
After pulling the latest main, did you run |
|
My dockerfile is based on the one provided in this repo, so the command is Also, I played around with the IK solver and noticed that when specifying the problem shape incorrectly (e.g. having a target position of shape I restarted the docker container and it works again (if the shapes are correct of course). |
If it’s not a bug, please use discussions: https://github.com/NVlabs/curobo/discussions
Please provide the below information in addition to your issue:
I ran the simple_stacking.py code and encountered a CUDA error when I changed it to my own robot.
`[0.040s] [ext: omni.kit.async_engine-0.0.0] startup
[0.220s] [ext: omni.activity.core-1.0.1] startup
[0.222s] [ext: omni.assets.plugins-0.0.0] startup
[0.223s] [ext: omni.stats-0.0.0] startup
[0.223s] [ext: omni.client-1.0.1] startup
[0.228s] [ext: omni.activity.profiler-1.0.2] startup
[0.229s] [ext: omni.gpu_foundation-0.0.0] startup
[0.234s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.235s] [ext: carb.windowing.plugins-1.0.0] startup
[0.239s] [ext: omni.kit.renderer.init-0.0.0] startup
|---------------------------------------------------------------------------------------------|
| Driver Version: 525.89.02 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 4090 | Yes: 0 | | 24564 MB | 10de | 0 |
| | | | | | 2684 | f723ef00.. |
| | | | | | 1 | |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 5.15.0-1051-intel-iotg
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: 13th Gen Intel(R) Core(TM) i7-13700KF | Cores: 16 | Logical: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 128653 | Free Memory: 90257
| Total Page/Swap (MB): 2047 | Free Page/Swap: 2018
|---------------------------------------------------------------------------------------------|
[0.641s] [ext: omni.kit.pipapi-0.0.0] startup
[0.642s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.642s] [ext: omni.pip.compute-1.2.0] startup
[0.642s] [ext: omni.pip.cloud-1.0.1] startup
[0.643s] [ext: omni.isaac.core_archive-2.2.1] startup
[0.643s] [ext: omni.pip.torch-2_0_1-2.0.2] startup
[0.664s] [ext: omni.kit.telemetry-0.5.0] startup
[0.680s] [ext: omni.kit.loop-isaac-1.1.0] startup
[0.681s] [ext: omni.kit.test-0.0.0] startup
[0.682s] [ext: omni.isaac.ml_archive-1.1.3] startup
[0.682s] [ext: omni.mtlx-0.1.0] startup
[0.682s] [ext: omni.usd.config-1.0.3] startup
[0.685s] [ext: omni.gpucompute.plugins-0.0.0] startup
[0.685s] [ext: omni.usd.libs-1.0.0] startup
[0.771s] [ext: omni.appwindow-1.1.5] startup
[1.135s] [ext: omni.kit.renderer.core-0.0.0] startup
[1.883s] [ext: omni.kit.renderer.capture-0.0.0] startup
[1.885s] [ext: omni.kit.renderer.imgui-0.0.0] startup
[1.931s] [ext: carb.audio-0.1.0] startup
[1.952s] [ext: omni.ui-2.18.5] startup
[1.960s] [ext: omni.uiaudio-1.0.0] startup
[1.961s] [ext: omni.kit.mainwindow-1.0.1] startup
[1.961s] [ext: omni.kit.uiapp-0.0.0] startup
[1.961s] [ext: omni.usd.schema.physics-0.0.0] startup
[2.054s] [ext: omni.usd.schema.audio-0.0.0] startup
[2.058s] [ext: omni.usd.schema.forcefield-0.0.0] startup
[2.062s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[2.066s] [ext: omni.usd.schema.scene.visualization-2.0.2] startup
[2.066s] [ext: omni.usd.schema.semantics-0.0.0] startup
[2.072s] [ext: omni.usd.schema.physx-0.0.0] startup
[2.091s] [ext: omni.usd.schema.anim-0.0.0] startup
[2.103s] [ext: omni.anim.navigation.schema-105.2.0] startup
[2.106s] [ext: omni.anim.graph.schema-105.2.2] startup
[2.111s] [ext: omni.usd.schema.isaac-2.0.1] startup
[2.116s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[2.118s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[2.121s] [ext: omni.resourcemonitor-105.0.0] startup
[2.122s] [ext: omni.timeline-1.0.9] startup
[2.124s] [ext: omni.usd_resolver-1.0.1] startup
[2.127s] [ext: omni.kit.window.popup_dialog-2.0.23] startup
[2.130s] [ext: omni.graph.exec-0.3.0] startup
[2.130s] [ext: omni.kit.actions.core-1.0.0] startup
[2.131s] [ext: omni.usd.core-1.1.8] startup
[2.134s] [ext: omni.kit.widget.nucleus_connector-1.1.4] startup
[2.136s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[2.136s] [ext: omni.kit.exec.core-0.5.0] startup
[2.138s] [ext: omni.kit.commands-1.4.6] startup
[2.140s] [ext: omni.hydra.usdrt_delegate-7.2.34] startup
[2.150s] [ext: omni.hydra.scene_delegate-0.3.2] startup
[2.156s] [ext: omni.usd-1.10.18] startup
[2.198s] [ext: omni.kit.widget.versioning-1.4.6] startup
[2.200s] [ext: omni.kit.helper.file_utils-0.1.6] startup
[2.214s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[2.215s] [ext: omni.kit.widget.filebrowser-2.3.35] startup
[2.217s] [ext: omni.kit.search_core-1.0.5] startup
[2.218s] [ext: omni.kit.widget.path_field-2.0.8] startup
[2.218s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[2.218s] [ext: omni.kit.widget.browser_bar-2.0.9] startup
[2.219s] [ext: omni.kit.collaboration.telemetry-1.0.0] startup
[2.220s] [ext: omni.kit.menu.utils-1.5.6] startup
[2.225s] [ext: omni.kit.collaboration.channel_manager-1.0.11] startup
[2.226s] [ext: omni.ui.scene-1.7.0] startup
[2.229s] [ext: omni.kit.notification_manager-1.0.6] startup
[2.230s] [ext: omni.kit.clipboard-1.0.3] startup
[2.231s] [ext: omni.kit.widget.options_menu-1.0.5] startup
[2.232s] [ext: omni.kit.usd.layers-2.1.27] startup
[2.238s] [ext: omni.kit.window.filepicker-2.10.13] startup
[2.244s] [ext: omni.kit.widget.filter-1.1.2] startup
[2.244s] [ext: omni.kit.menu.create-1.0.11] startup
[2.245s] [ext: omni.kit.window.file_exporter-1.0.22] startup
[2.247s] [ext: omni.kit.hotkeys.core-1.3.0] startup
[2.249s] [ext: omni.kit.context_menu-1.6.8] startup
[2.250s] [ext: omni.kit.widget.stage-2.9.6] startup
[2.255s] [ext: omni.kit.primitive.mesh-1.0.15] startup
[2.257s] [ext: omni.kit.usd_undo-0.1.2] startup
[2.258s] [ext: omni.kit.stage_templates-1.1.20] startup
[2.258s] [ext: omni.kit.widget.searchfield-1.1.3] startup
[2.259s] [ext: omni.kit.widget.highlight_label-1.0.1] startup
[2.259s] [ext: omni.kit.window.file_importer-1.0.22] startup
[2.260s] [ext: omni.kit.window.property-1.9.6] startup
[2.261s] [ext: omni.ui_query-1.1.1] startup
[2.262s] [ext: omni.inspect-1.0.1] startup
[2.263s] [ext: usdrt.scenegraph-7.2.34] startup
[2.291s] [ext: omni.graph.tools-1.41.3] startup
[2.301s] [ext: omni.graph.core-2.139.12] startup
[2.304s] [ext: omni.kit.window.file-1.3.44] startup
[2.305s] [ext: omni.graph-1.106.1] startup
[2.332s] [ext: omni.kvdb-0.0.0] startup
[2.334s] [ext: omni.convexdecomposition-105.1.12-5.1] startup
[2.338s] [ext: omni.localcache-0.0.0] startup
[2.340s] [ext: omni.iray.libs-0.0.0] startup
[2.343s] [ext: omni.physx.foundation-105.1.12-5.1] startup
[2.343s] [ext: omni.usdphysics-105.1.12-5.1] startup
[2.346s] [ext: omni.mdl.neuraylib-0.2.0] startup
[2.347s] [ext: omni.isaac.version-1.0.3] startup
[2.348s] [ext: omni.physx.cooking-105.1.12-5.1] startup
[2.355s] [ext: omni.mdl-0.2.1] startup
[2.367s] [ext: omni.physics.tensors-0.1.0] startup
[2.373s] [ext: omni.physx-105.1.12-5.1] startup
2024-04-19 05:29:57 [2,374ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
[2.383s] [ext: omni.kit.material.library-1.3.41] startup
[2.387s] [ext: omni.volume-0.4.0] startup
[2.389s] [ext: omni.physx.tensors-0.1.0] startup
[2.393s] [ext: omni.kit.numpy.common-0.1.2] startup
[2.395s] [ext: omni.hydra.rtx-0.1.0] startup
[2.402s] [ext: omni.warp.core-1.0.0-beta.2] startup
Warp 1.0.0-beta.2 initialized:
CUDA Toolkit: 11.5, Driver: 12.0
Devices:
"cpu" | x86_64
"cuda:0" | NVIDIA GeForce RTX 4090 (sm_89)
Kernel cache: /home/lily/.cache/warp/1.0.0-beta.2
[2.487s] [ext: omni.isaac.dynamic_control-1.2.6] startup
[2.495s] [ext: omni.debugdraw-0.1.2] startup
[2.499s] [ext: omni.isaac.core-3.3.2] startup
[2.667s] [ext: omni.hydra.pxr-1.1.6] startup
[2.672s] [ext: omni.kit.collaboration.presence_layer-1.0.8] startup
[2.673s] [ext: omni.kit.widget.prompt-1.0.6] startup
[2.674s] [ext: omni.kit.hydra_texture-1.1.12] startup
[2.679s] [ext: omni.hydra.engine.stats-1.0.1] startup
[2.683s] [ext: omni.kit.viewport.legacy_gizmos-1.0.14] startup
[2.687s] [ext: omni.kit.widget.viewport-105.1.7] startup
[2.690s] [ext: omni.kit.widget.searchable_combobox-1.0.5] startup
[2.691s] [ext: omni.kit.window.drop_support-1.0.1] startup
[2.692s] [ext: omni.kit.viewport.registry-104.0.5] startup
[2.693s] [ext: omni.kit.widget.settings-1.0.6] startup
[2.694s] [ext: omni.kit.window.content_browser_registry-0.0.1] startup
[2.695s] [ext: omni.kit.viewport.window-105.1.12] startup
[2.709s] [ext: omni.kit.window.preferences-1.3.23] startup
[2.714s] [ext: omni.kit.widget.graph-1.8.4] startup
[2.717s] [ext: omni.kit.window.content_browser-2.7.17] startup
[2.725s] [ext: omni.kit.viewport.utility-1.0.16] startup
[2.726s] [ext: omni.kit.widget.live_session_management-1.2.22] startup
[2.729s] [ext: omni.kit.window.extensions-1.3.0] startup
[2.733s] [ext: omni.kit.widget.text_editor-1.0.2] startup
[2.735s] [ext: omni.kit.property.usd-3.21.15] startup
[2.742s] [ext: omni.kit.manipulator.viewport-104.0.9] startup
[2.743s] [ext: omni.graph.ui-1.48.0] startup
[2.756s] [ext: omni.kit.ui_test-1.2.13] startup
[2.757s] [ext: omni.graph.visualization.nodes-2.1.1] startup
[2.768s] [ext: omni.graph.action-1.55.1] startup
[2.797s] [ext: omni.isaac.ui-0.13.5] startup
[2.803s] [ext: omni.isaac.surface_gripper-0.7.0] startup
[2.812s] [ext: omni.isaac.lula-2.1.0] startup
[2.817s] [ext: omni.kit.graph.delegate.default-1.1.1] startup
[2.818s] [ext: omni.isaac.manipulators-1.1.0] startup
[2.822s] [ext: omni.isaac.motion_generation-6.1.0] startup
[2.827s] [ext: omni.kit.graph.editor.core-1.5.1] startup
[2.829s] [ext: omni.kit.graph.delegate.modern-1.9.1] startup
[2.830s] [ext: omni.isaac.franka-0.4.0] startup
[2.832s] [ext: omni.kit.graph.widget.variables-2.1.0] startup
[2.834s] [ext: omni.kit.graph.usd.commands-1.3.0] startup
[2.835s] [ext: omni.isaac.universal_robots-0.3.3] startup
[2.837s] [ext: omni.isaac.cortex-0.3.5] startup
[2.838s] [ext: omni.graph.window.core-1.84.0] startup
[2.846s] [ext: omni.isaac.dofbot-0.3.1] startup
[2.848s] [ext: omni.isaac.cortex.sample_behaviors-1.0.4] startup
[2.850s] [ext: omni.graph.window.generic-1.9.3] startup
[2.854s] [ext: omni.isaac.kit-1.4.7] startup
[2.855s] [ext: omni.graph.window.action-1.11.2] startup
[2.859s] [ext: omni.kit.widget.zoombar-1.0.5] startup
[2.860s] [ext: omni.graph.image.core-0.1.0] startup
[2.861s] [ext: omni.kit.window.cursor-1.1.2] startup
[2.862s] [ext: omni.kit.browser.core-2.3.10] startup
[2.865s] [ext: omni.kit.widget.spinner-1.0.6] startup
[2.866s] [ext: omni.graph.nodes-1.106.3] startup
[2.992s] [ext: omni.kit.widget.material_preview-1.0.8] startup
[2.994s] [ext: omni.kit.browser.folder.core-1.9.7] startup
[2.997s] [ext: omni.kit.viewport.menubar.core-105.0.19] startup
[3.009s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[3.011s] [ext: omni.kit.window.material_graph-1.8.11] startup
[3.026s] [ext: omni.kit.browser.sample-1.4.6] startup
[3.029s] [ext: omni.syntheticdata-0.4.10] startup
[3.050s] [ext: omni.kit.livestream.core-2.3.0] startup
[3.052s] [ext: omni.kit.streamsdk.plugins-2.5.2] startup
[3.052s] [ext: omni.warp-1.0.0-beta.2] startup
[3.100s] [ext: omni.graph.scriptnode-1.1.6] startup
[3.106s] [ext: omni.command.usd-1.0.2] startup
[3.107s] [ext: omni.kit.livestream.native-2.4.0] startup
Active user not found. Using default user [kiosk]Streaming server started.
[3.190s] [ext: omni.replicator.core-1.10.20] startup
2024-04-19 05:29:58 [3,198ms] [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
[3.319s] [ext: omni.kit.viewport.menubar.lighting-105.0.7] startup
[3.322s] [ext: omni.isaac.urdf-1.1.0] startup
2024-04-19 05:29:58 [3,315ms] [Warning] [omni.isaac.urdf] The extension omni.isaac.urdf has been deprecated. Please use omni.importer.urdf instead.
[3.323s] [ext: omni.isaac.core_nodes-1.7.1] startup
[3.343s] [ext: omni.isaac.cloner-0.7.2] startup
[3.344s] [ext: omni.isaac.utils-0.6.0] startup
[3.352s] [ext: omni.isaac.wheeled_robots-0.9.0] startup
[3.366s] [ext: omni.isaac.gym-0.10.0] startup
[3.368s] [ext: omni.kit.selection-0.1.3] startup
[3.369s] [ext: omni.kit.menu.common-1.1.4] startup
[3.371s] [ext: omni.kit.widget.layers-1.6.32] startup
[3.378s] [ext: omni.kit.menu.edit-1.1.14] startup
[3.381s] [ext: omni.kit.menu.file-1.1.7] startup
[3.383s] [ext: omni.kit.property.layer-1.1.5] startup
[3.385s] [ext: semantics.schema.editor-0.3.4] startup
[3.387s] [ext: omni.kit.property.camera-1.0.5] startup
[3.388s] [ext: omni.kit.property.audio-1.0.9] startup
[3.392s] [ext: omni.kit.profiler.window-2.1.2] startup
2024-04-19 05:29:58 [3,456ms] [Warning] [omni.kit.profiler.window] remove _SpanInstance.lt and use insort 'key' arg instead
[3.466s] [ext: omni.kit.widget.stage_icons-1.0.4] startup
[3.467s] [ext: omni.hydra.scene_api-0.1.2] startup
[3.474s] [ext: omni.kit.property.light-1.0.8] startup
[3.476s] [ext: omni.kit.property.skel-1.0.4] startup
[3.478s] [ext: omni.kit.window.stage-2.4.4] startup
[3.481s] [ext: omni.kit.property.transform-1.3.10] startup
[3.485s] [ext: omni.kit.property.render-1.1.1] startup
[3.486s] [ext: omni.kit.property.material-1.8.36] startup
[3.489s] [ext: omni.kit.property.geometry-1.2.11] startup
[3.492s] [ext: omni.isaac.debug_draw-0.6.1] startup
2024-04-19 05:29:58 [3,489ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-04-19 05:29:58 [3,489ms] [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
[3.500s] [ext: omni.kit.menu.stage-1.2.3] startup
[3.502s] [ext: omni.kit.property.bundle-1.2.9] startup
[3.503s] [ext: omni.kit.property.isaac-0.2.2] startup
[3.505s] [ext: omni.isaac.occupancy_map-0.3.2] startup
[3.522s] [ext: omni.isaac.mjcf-1.0.0] startup
2024-04-19 05:29:58 [3,515ms] [Warning] [omni.isaac.mjcf] The extension omni.isaac.mjcf has been deprecated. Please use omni.importer.mjcf instead.
[3.522s] [ext: omni.kit.viewport.actions-105.1.6] startup
[3.526s] [ext: omni.kit.widget.toolbar-1.5.6] startup
[3.532s] [ext: omni.kit.raycast.query-1.0.1] startup
[3.539s] [ext: omni.kit.manipulator.transform-104.6.15] startup
[3.541s] [ext: omni.rtx.window.settings-0.6.9] startup
[3.545s] [ext: omni.kit.viewport.menubar.display-105.1.0] startup
[3.547s] [ext: omni.kit.manipulator.tool.snap-1.4.2] startup
[3.552s] [ext: omni.hydra.pxr.settings-1.0.5] startup
[3.554s] [ext: omni.activity.pump-1.0.0] startup
[3.556s] [ext: omni.physx.commands-105.1.12-5.1] startup
[3.558s] [ext: omni.usdphysics.ui-105.1.12-5.1] startup
[3.583s] [ext: omni.kit.widget.live-2.1.6] startup
[3.585s] [ext: omni.kit.viewport.pxr-104.0.1] startup
[3.585s] [ext: omni.kit.viewport.ready-1.0.5] startup
[3.586s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[3.586s] [ext: omni.physx.ui-105.1.12-5.1] startup
[3.609s] [ext: omni.kit.widget.cache_indicator-2.0.8] startup
[3.611s] [ext: usdrt.gf-1.0.2] startup
[3.611s] [ext: omni.kit.manipulator.selector-1.0.1] startup
[3.613s] [ext: omni.kit.window.commands-0.2.4] startup
[3.614s] [ext: omni.kit.window.console-0.2.10] startup
[3.623s] [ext: omni.kit.property.physx-0.1.0] startup
[3.671s] [ext: omni.kit.actions.window-1.1.0] startup
[3.674s] [ext: omni.physx.demos-105.1.12-5.1] startup
[3.677s] [ext: omni.kit.manipulator.prim-105.0.11] startup
[3.682s] [ext: omni.kit.window.toolbar-1.5.4] startup
[3.684s] [ext: omni.kit.manipulator.selection-104.0.8] startup
[3.686s] [ext: omni.kit.window.script_editor-1.7.4] startup
[3.688s] [ext: omni.kit.hotkeys.window-1.4.1] startup
[3.691s] [ext: omni.physx.camera-105.1.12-5.1] startup
[3.698s] [ext: omni.physx.cct-105.1.12-5.1] startup
[3.709s] [ext: omni.physx.graph-105.1.12-5.1] startup
[3.719s] [ext: omni.physx.supportui-105.1.12-5.1] startup
[3.731s] [ext: omni.physx.vehicle-105.1.12-5.1] startup
[3.741s] [ext: omni.kit.stage_column.payload-1.0.12] startup
[3.743s] [ext: omni.graph.ui_nodes-1.5.10] startup
[3.766s] [ext: omni.sensors.nv.common-1000.0.0-isaacsim.2023.1.1] startup
[3.773s] [ext: omni.sensors.nv.materials-1000.0.0-isaacsim.2023.1.1] startup
[3.774s] [ext: omni.sensors.nv.wpm-1000.0.0-isaacsim.2023.1.1] startup
[3.775s] [ext: omni.kit.window.status_bar-0.1.6] startup
[3.779s] [ext: omni.physx.bundle-105.1.12-5.1] startup
[3.779s] [ext: omni.graph.bundle.action-2.0.4] startup
[3.779s] [ext: omni.isaac.range_sensor-2.0.2] startup
[3.813s] [ext: omni.sensors.nv.lidar-1000.0.1-isaacsim.2023.1.1] startup
[3.818s] [ext: omni.sensors.nv.radar-1000.0.0-isaacsim.2023.1.1] startup
[3.822s] [ext: omni.kit.window.title-1.1.3] startup
[3.823s] [ext: omni.replicator.composer-1.2.17] startup
[3.838s] [ext: omni.replicator.isaac-1.11.3] startup
[3.857s] [ext: omni.kit.manipulator.camera-105.0.4] startup
[3.860s] [ext: omni.kit.viewport.menubar.camera-105.1.9] startup
[3.863s] [ext: omni.isaac.sensor-9.11.1] startup
[3.917s] [ext: omni.kit.viewport.menubar.render-105.1.1] startup
[3.919s] [ext: omni.kit.viewport.menubar.settings-105.0.9] startup
[3.923s] [ext: omni.replicator.replicator_yaml-2.0.4] startup
[3.931s] [ext: omni.rtx.settings.core-0.5.13] startup
[3.934s] [ext: omni.kit.stage_column.variant-1.0.13] startup
[3.936s] [ext: omni.isaac.quadruped-1.4.1] startup
[3.945s] [ext: semantics.schema.property-1.0.2] startup
[3.947s] [ext: omni.kit.viewport.bundle-104.0.1] startup
[3.947s] [ext: omni.isaac.scene_blox-0.1.0] startup
[3.949s] [ext: omni.importer.urdf-1.6.1] startup
[3.982s] [ext: omni.importer.mjcf-1.1.0] startup
[3.990s] [ext: omni.kit.window.stats-0.1.4] startup
[3.991s] [ext: omni.isaac.sim.python-2023.1.1] startup
[3.992s] Simulation App Starting
2024-04-19 05:29:59 [4,002ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/ecoMode/enabled has been requested but is not available
2024-04-19 05:29:59 [4,002ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/directLighting/enabled has been requested but is not available
2024-04-19 05:29:59 [4,002ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/reflections/enabled has been requested but is not available
2024-04-19 05:29:59 [4,002ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/translucency/enabled has been requested but is not available
2024-04-19 05:29:59 [4,002ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/raytracing/subsurface/enabled has been requested but is not available
2024-04-19 05:29:59 [4,003ms] [Warning] [omni.kit.widget.settings.settings_model] a value for setting /rtx/caustics/enabled has been requested but is not available
2024-04-19 05:30:00 [4,889ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-04-19 05:30:00 [4,889ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
[6.010s] app ready
2024-04-19 05:30:01 [6,464ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-04-19 05:30:01 [6,464ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
[6.690s] RTX ready
[6.853s] Simulation App Startup Complete
my_task = MultiModalStacking()
my_world.add_task(my_task)
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_5/visuals' has corrupted data in primvar 'st_2': buffer size 1602 doesn't match expected size 92748 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_5/visuals' has corrupted data in primvar 'st_1': buffer size 12261 doesn't match expected size 92748 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_5/visuals' has corrupted data in primvar 'st': buffer size 78885 doesn't match expected size 92748 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_4/visuals' has corrupted data in primvar 'st_2': buffer size 1602 doesn't match expected size 93852 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_3/visuals' has corrupted data in primvar 'st_3': buffer size 1116 doesn't match expected size 56562 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_4/visuals' has corrupted data in primvar 'st_1': buffer size 13389 doesn't match expected size 93852 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_4/visuals' has corrupted data in primvar 'st': buffer size 78861 doesn't match expected size 93852 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_3/visuals' has corrupted data in primvar 'st_2': buffer size 10962 doesn't match expected size 56562 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_3/visuals' has corrupted data in primvar 'st_1': buffer size 6339 doesn't match expected size 56562 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_3/visuals' has corrupted data in primvar 'st': buffer size 38145 doesn't match expected size 56562 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_1/visuals' has corrupted data in primvar 'st_2': buffer size 1872 doesn't match expected size 98154 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_1/visuals' has corrupted data in primvar 'st_1': buffer size 14124 doesn't match expected size 98154 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_1/visuals' has corrupted data in primvar 'st': buffer size 82158 doesn't match expected size 98154 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_6/visuals' has corrupted data in primvar 'st_3': buffer size 144 doesn't match expected size 109944 in faceVarying primvars
2024-04-19 05:30:05 [9,929ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_2/visuals' has corrupted data in primvar 'st_2': buffer size 3456 doesn't match expected size 116106 in faceVarying primvars
2024-04-19 05:30:05 [9,930ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_6/visuals' has corrupted data in primvar 'st_2': buffer size 1434 doesn't match expected size 109944 in faceVarying primvars
2024-04-19 05:30:05 [9,930ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_2/visuals' has corrupted data in primvar 'st_1': buffer size 18303 doesn't match expected size 116106 in faceVarying primvars
2024-04-19 05:30:05 [9,930ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_6/visuals' has corrupted data in primvar 'st_1': buffer size 555 doesn't match expected size 109944 in faceVarying primvars
2024-04-19 05:30:05 [9,930ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_2/visuals' has corrupted data in primvar 'st': buffer size 94347 doesn't match expected size 116106 in faceVarying primvars
2024-04-19 05:30:05 [9,930ms] [Warning] [omni.hydra] Mesh '/World/tm12/link_6/visuals' has corrupted data in primvar 'st': buffer size 107811 doesn't match expected size 109944 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_10': buffer size 1770 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_11': buffer size 378 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_7': buffer size 1722 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_6': buffer size 10452 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_9': buffer size 54930 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_4': buffer size 28728 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_8': buffer size 12117 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_3': buffer size 518187 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_5': buffer size 36216 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_2': buffer size 1515 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,931ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st_1': buffer size 30030 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [9,932ms] [Warning] [omni.hydra] Mesh '/World/tm12/epick_base/visuals' has corrupted data in primvar 'st': buffer size 8556 doesn't match expected size 704601 in faceVarying primvars
2024-04-19 05:30:05 [10,068ms] [Warning] [omni.physx.plugin] The rigid body at /World/tm12/base has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-04-19 05:30:05 [10,068ms] [Warning] [omni.physx.plugin] The rigid body at /World/tm12/epick_ee has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-04-19 05:30:05 [10,068ms] [Warning] [omni.physx.plugin] The rigid body at /World/tm12/flange_link has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-04-19 05:30:05 [10,068ms] [Warning] [omni.physx.plugin] The rigid body at /World/tm12/world has a possibly invalid inertia tensor of {1.0, 1.0, 1.0} and a negative mass, small sphere approximated inertia was used. Either specify correct values in the mass properties, or add collider(s) to any shape(s) that you wish to automatically compute mass properties for.
2024-04-19 05:30:05 [10,069ms] [Warning] [omni.physx.plugin] Detected an articulation at /World/tm12 with more than 4 velocity iterations being added to a TGS scene.The related behavior changed recently, please consult the changelog. This warning will only print once.
warming up...
Start simulation...
TGS 124 4
False
(array([5.729578e+08, 5.729578e+08, 5.729578e+08, 5.729578e+08,
5.729578e+08, 5.729578e+08], dtype=float32), array([5729578., 5729578., 5729578., 5729578., 5729578., 5729578.],
dtype=float32))
[353. 353. 157. 54. 54. 54.]
Updated gains:
(array([5.729578e+08, 5.729578e+08, 5.729578e+08, 5.729578e+08,
5.729578e+08, 5.729578e+08], dtype=float32), array([5729578., 5729578., 5729578., 5729578., 5729578., 5729578.],
dtype=float32))
[353. 353. 157. 54. 54. 54.]
[20.885s] [ext: omni.kit.asset_converter-2.0.18] startup
[20.907s] [ext: omni.kit.tool.asset_importer-2.4.28] startup
[20.951s] [ext: omni.isaac.asset_browser-0.4.3] startup
2024-04-19 05:30:25 [30,534ms] [Warning] [curobo] Creating new Mesh cache: 16
Traceback (most recent call last):
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/examples/isaac_sim/simple_stacking_tm12.py", line 545, in
art_action = my_controller.forward(sim_js, my_tm12.dof_names)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/examples/isaac_sim/simple_stacking_tm12.py", line 262, in forward
result = self.plan(ee_translation_goal, ee_orientation_goal, sim_js, js_names)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/examples/isaac_sim/simple_stacking_tm12.py", line 229, in plan
result = self.motion_gen.plan_single(cu_js.unsqueeze(0), ik_goal, self.plan_config.clone())
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1395, in plan_single
result = self._plan_attempts(
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1193, in _plan_attempts
result = self._plan_from_solve_state(
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1511, in _plan_from_solve_state
ik_result = self._solve_ik_from_solve_state(
File "/home/lily/.local/share/ov/pkg/isaac_sim-2023.1.1/kit/python/lib/python3.10/contextlib.py", line 79, in inner
return func(*args, **kwds)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/motion_gen.py", line 1020, in _solve_ik_from_solve_state
ik_result = self.ik_solver.solve_any(
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/ik_solver.py", line 1160, in solve_any
return self.solve_single(
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/ik_solver.py", line 654, in solve_single
return self._solve_from_solve_state(
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/reacher/ik_solver.py", line 1043, in _solve_from_solve_state
result = self.solver.solve(goal_buffer, coord_position_seed)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/wrap_base.py", line 147, in solve
act_seq = self.optimize(seed, shift_steps=0)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/wrap/wrap_base.py", line 71, in optimize
act_seq = opt.optimize(act_seq, shift_steps)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/opt/opt_base.py", line 170, in optimize
out = self._optimize(opt_tensor, shift_steps, n_iters)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/opt/newton/newton_base.py", line 146, in _optimize
best_q, best_cost, q, grad_q = self._call_opt_iters_graph(q, grad_q)
File "/home/lily/Desktop/20230605_Job_Techman_Project/01_Omniverse_IsaacSim/02_Robotics_Simulator/2024Q2_curobo/curobo/src/curobo/opt/newton/newton_base.py", line 521, in _call_opt_iters_graph
self._cu_opt_q.clone(),
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.The text was updated successfully, but these errors were encountered: