tester-occ-7.4.0 #300

Andrei-Pozolotin · 2020-03-20T18:58:14Z

this is a summary of changes needed to start a branch
to track migration towards occ-7.4.0
currently, there are 40 failing tests:
tool/pytest_todo.md
for context, see:

Investigate PythonOCC based on OCCT7.4
Investigate PythonOCC based on OCCT7.4 #226
OCCT 7.4.0: Undefined references, again
OCCT 7.4.0: Undefined references, again tpaviot/pythonocc-core#795

jmwright · 2020-03-21T23:30:50Z

@adam-urbanczyk Can correct me, but I believe all tests pass on the OCP branch, but CI hasn't been updated.

https://github.com/CadQuery/cadquery/tree/OCP

I like the PythonOCC project, but I think @adam-urbanczyk made a compelling case for having our own bindings here: CadQuery/OCP#4 (comment)

When CQ was originally based on FreeCAD, we started hitting the limits of what their API exposed to us. PythonOCC is pretty complete, but Adam is using some more modern binding technologies to get us close to a 1:1 set of output bindings that are generated in a much more automated way than I've seen from other projects. That makes updates for new versions of OCCT very fast too.

We're already going through a disruptive transition away from FreeCAD, so my thought is that we might as well go through the extra pain of switching to custom, auto-generated bindings and get it over with. I believe that will set us up better for the future. That's my two cents. The discussion is still open though.

jmwright · 2020-03-21T23:42:09Z

I forgot that @adam-urbanczyk just did an alpha release of OCP with builds for the three major OSes.

https://github.com/CadQuery/OCP/releases/tag/7.4-alpha

That makes it easier to get started with OCP.

Andrei-Pozolotin · 2020-03-21T23:45:09Z

@jmwright Jeremy:
it seems that Adam is sending mixed messages to the troops:

#183 (comment)

Q:
does it mean OCP will replace OCC some time soon?
A:
That is possible, however should be transparent from the user point of view. Both PythonOCC and OCP are rather thin warappers of OCCT. Ath the end of the day the goal is to have the latest OCCT version as a basis and be future proof w.r.t. coming OCCT releases.

to me that sounds like he wants to support both OCP and P-OCC, no? :-)

instead, could have just said straight "NO" to P-OCC. oh, well... :-)

Andrei-Pozolotin · 2020-03-21T23:58:06Z

ok, I finally got to the source of my confusion:

https://github.com/CadQuery/cadquery/tree/master
is based on P-OCC, which is used by 1000's
builds fine, and is current

https://github.com/CadQuery/cadquery/tree/OCP
is based on experimental tech (read: no one uses)
does not build, and is behind the P-OCC branch

then Adam says he likes both. well then I say: I like P-OCC :-)

jmwright · 2020-03-22T00:11:36Z

@Andrei-Pozolotin I think he and I both are sending mixed signals because we don't want to force a decision yet. I don't think the community has had time to absorb what OCP is, let alone build any consensus around the decision. As core developers we can force changes quickly, but we've got a great community that offers good insights. I personally want to provide them some time. If nobody has any thoughts that's fine, but at least there was a chance given.

My personal preference as a core developer would be to make OCP the default. If you and/or someone else also wanted to help maintain PythonOCC support, I'm not going to tell you "no" unless there is a good reason to (i.e. it would make the codebase unmaintainable to support both). Also, if somebody can make a great case for making PythonOCC the default instead of OCP, I'm willing to listen.

adam-urbanczyk · 2020-03-22T00:29:28Z

Let me be clear @Andrei-Pozolotin - I think the best decision would be to go on and use our own (i.e. OCP) wrapper for CadQuery. So far it has more streamlined (and IMO more future proof) binding generation technology (clang+pybind11) that will allow to quickly adapt to new releases of OCCT. The other thing is that PythonOCC does not seem to offer bindings for all modules of OCCT. I want to still reach out to the PythonOCC community to check if my observations are correct.

Your perception of giving mixed messages probably stems for the fact that we did not make this decision yet. I don't think that there will be bandwidth from the core team to support both wrappers. What I meant in the comment was that whichever wrapper is basis of CQ should not matter from the user point of view - it will be based on OCCT as the CAD kernel.

I really appreciate the PR request, it is good to have testing possibilities for all options. I have a question though: what are you trying to achieve, what difference does it make to you which wrapper CQ is going to use?

Andrei-Pozolotin · 2020-03-22T04:04:19Z

@jmwright Jeremy @adam-urbanczyk Adam

Thank you guys for taking the time to respond and explain, here is my story :-)

what are you trying to achieve

I was looking for some time for a python-centric solution for parametric design
I quickly discovered CQ as viable choice - architecturally speaking, that is
except that CQ "was never ready", "just a nice toy", "great demo gig", etc
finally, recently I decide to take a dive, built internally quick prototype, and:
the first thing I discover that CQ2 does not scale at all:
simple parametric 15x15 grid union takes 3 minutes on 3 GHz CPU :-)
which leads me to How to merge shape? #183 (comment) Adam saying I should try 7.4.0 for performance
then I discover I can not build OCP, but with PythonOCC I have some experience
which results in the useless ordeal described above that ends in this tester-occ-7.4.0 #300

what difference does it make to you which wrapper CQ is going to use

As any "normal user" would answer:
I do not care, I just need something that works
which at the moment - nothing does (either can not build or can not perform)

Now, please listen to yourselves:

I think he and I both are sending mixed signals
I personally want to provide them some time
My personal preference as a core developer would be to make OCP the default
I think the best decision would be to go on and use our own
probably stems for the fact that we did not make this decision yet

Guys, stop deceiving yourselves - the decision is already made.
If you can not see that, please allow me to make one for you: OCP.

And in the very unlikely event anyone will ask for PythonOCC,
you can deal with that later by introducing additional api indirection layer.

Thanks again!

adam-urbanczyk · 2020-03-22T17:01:41Z

@Andrei-Pozolotin you don't need to build OCP yourself, just pull it form the CI here or from the releases here https://github.com/CadQuery/OCP/releases/tag/7.4-alpha (py3.6 only for now).

FYI: I just did merge master into OCP.

Andrei-Pozolotin · 2020-03-22T17:47:51Z

@adam-urbanczyk Adam:

great news, thank you.
what does it take to produce a cadquery-dev channel
based on that OCP pipeline, for "normal users", which "just works" ? i.e. :

conda install -y -c conda-forge -c cadquery-dev cadquery

adam-urbanczyk · 2020-03-26T08:03:05Z

@Andrei-Pozolotin it is just "work". I will eventually provide conda packages for OCP.

Andrei-Pozolotin · 2020-03-26T15:25:31Z

great, thank you, looking forward to that

Andrei-Pozolotin · 2020-04-04T19:27:54Z

@adam-urbanczyk, @jmwright, update:

still facing a performance problem after switch to `cq2/ocp`

I published ocp-7.4.0, cq2/ocp, cq-parts/ocp here:
https://anaconda.org/random-builder/repo

ocp-7.4.0 from OCP/releases/tag/7.4-alpha
via random-builder/design_cadquery_pywrap@34c8d5b
https://anaconda.org/random-builder/cadquery-pyocct
cq2/ocp from cadquery/tree/OCP
via random-builder@3d54547
https://anaconda.org/random-builder/cadquery-arkon
cq-parts/ocp from jmwright/cqparts
via random-builder/design_cadquery_parts@5e7b89d
https://anaconda.org/random-builder/cq-parts-bundle

now a "regular user" can install all of the above:

conda install -y \
    -c conda-forge \
    -c random-builder \
    cadquery-arkon \
    cq-parts-bundle \


Remove all packages in environment /home/work/apps/miniconda/envs/creator-model:

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/work/apps/miniconda/envs/creator-model

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  ca-certificates    pkgs/main/linux-64::ca-certificates-2020.1.1-0
  certifi            pkgs/main/linux-64::certifi-2019.11.28-py36_1
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.33.1-h53a641e_7
  libedit            pkgs/main/linux-64::libedit-3.1.20181209-hc058e9b_0
  libffi             pkgs/main/linux-64::libffi-3.2.1-hd88cf55_4
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.1.0-hdf63c60_0
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.1.0-hdf63c60_0
  ncurses            pkgs/main/linux-64::ncurses-6.2-he6710b0_0
  openssl            pkgs/main/linux-64::openssl-1.1.1f-h7b6447c_0
  pip                pkgs/main/linux-64::pip-20.0.2-py36_1
  python             pkgs/main/linux-64::python-3.6.10-hcf32534_1
  readline           pkgs/main/linux-64::readline-8.0-h7b6447c_0
  setuptools         pkgs/main/linux-64::setuptools-46.1.3-py36_0
  sqlite             pkgs/main/linux-64::sqlite-3.31.1-h7b6447c_0
  tk                 pkgs/main/linux-64::tk-8.6.8-hbc83047_0
  wheel              pkgs/main/linux-64::wheel-0.34.2-py36_0
  xz                 pkgs/main/linux-64::xz-5.2.4-h14c3975_4
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7b6447c_3


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate creator-model
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

## Package Plan ##

  environment location: /home/work/apps/miniconda/envs/creator-model

  added / updated specs:
    - cadquery-arkon
    - cq-parts-bundle


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cq-parts-bundle-2.0.0      |            py3.6          74 KB  random-builder
    ------------------------------------------------------------
                                           Total:          74 KB

The following NEW packages will be INSTALLED:

  cadquery-arkon     random-builder/noarch::cadquery-arkon-2.0.0-py3.6
  cadquery-pyocct    random-builder/linux-64::cadquery-pyocct-7.4.0-py3.6
  cffi               conda-forge/linux-64::cffi-1.14.0-py36hd463f26_0
  chardet            conda-forge/linux-64::chardet-3.0.4-py36h9f0ad1d_1006
  cq-parts-bundle    random-builder/noarch::cq-parts-bundle-2.0.0-py3.6
  cryptography       conda-forge/linux-64::cryptography-2.8-py36h45558ae_2
  fontconfig         conda-forge/linux-64::fontconfig-2.13.1-h86ecdb6_1001
  freetype           conda-forge/linux-64::freetype-2.10.1-he06d7ca_0
  icu                conda-forge/linux-64::icu-64.2-he1b5a44_1
  idna               conda-forge/noarch::idna-2.9-py_1
  jinja2             conda-forge/noarch::jinja2-2.11.1-py_0
  libblas            conda-forge/linux-64::libblas-3.8.0-14_openblas
  libcblas           conda-forge/linux-64::libcblas-3.8.0-14_openblas
  libgfortran-ng     conda-forge/linux-64::libgfortran-ng-7.3.0-hdf63c60_5
  libiconv           conda-forge/linux-64::libiconv-1.15-h516909a_1006
  liblapack          conda-forge/linux-64::liblapack-3.8.0-14_openblas
  libopenblas        conda-forge/linux-64::libopenblas-0.3.7-h5ec1e0e_6
  libpng             conda-forge/linux-64::libpng-1.6.37-hed695b0_1
  libuuid            conda-forge/linux-64::libuuid-2.32.1-h14c3975_1000
  libxcb             conda-forge/linux-64::libxcb-1.13-h14c3975_1002
  libxml2            conda-forge/linux-64::libxml2-2.9.10-hee79883_0
  markupsafe         conda-forge/linux-64::markupsafe-1.1.1-py36h8c4c3a4_1
  numpy              conda-forge/linux-64::numpy-1.18.1-py36h7314795_1
  occt               conda-forge/linux-64::occt-7.4.0-h9121d39_3
  pthread-stubs      conda-forge/linux-64::pthread-stubs-0.4-h14c3975_1001
  pycparser          conda-forge/noarch::pycparser-2.20-py_0
  pyopenssl          conda-forge/noarch::pyopenssl-19.1.0-py_1
  pyparsing          conda-forge/noarch::pyparsing-2.4.6-py_0
  pysocks            conda-forge/linux-64::pysocks-1.7.1-py36h9f0ad1d_1
  python_abi         conda-forge/linux-64::python_abi-3.6-1_cp36m
  requests           conda-forge/noarch::requests-2.23.0-pyh8c360ce_2
  six                conda-forge/noarch::six-1.14.0-py_1
  tbb                conda-forge/linux-64::tbb-2020.1-hc9558a2_0
  tinydb             conda-forge/noarch::tinydb-3.15.2-py_0
  urllib3            conda-forge/linux-64::urllib3-1.25.7-py36h9f0ad1d_1
  xorg-kbproto       conda-forge/linux-64::xorg-kbproto-1.0.7-h14c3975_1002
  xorg-libice        conda-forge/linux-64::xorg-libice-1.0.10-h516909a_0
  xorg-libsm         conda-forge/linux-64::xorg-libsm-1.2.3-h84519dc_1000
  xorg-libx11        conda-forge/linux-64::xorg-libx11-1.6.9-h516909a_0
  xorg-libxau        conda-forge/linux-64::xorg-libxau-1.0.9-h14c3975_0
  xorg-libxdmcp      conda-forge/linux-64::xorg-libxdmcp-1.1.3-h516909a_0
  xorg-libxt         conda-forge/linux-64::xorg-libxt-1.2.0-h516909a_0
  xorg-xproto        conda-forge/linux-64::xorg-xproto-7.0.31-h14c3975_1007

The following packages will be SUPERSEDED by a higher-priority channel:

  ca-certificates     pkgs/main::ca-certificates-2020.1.1-0 --> conda-forge::ca-certificates-2019.11.28-hecc5488_0
  certifi              pkgs/main::certifi-2019.11.28-py36_1 --> conda-forge::certifi-2019.11.28-py36h9f0ad1d_1
  openssl              pkgs/main::openssl-1.1.1f-h7b6447c_0 --> conda-forge::openssl-1.1.1f-h516909a_0



Downloading and Extracting Packages

cq-parts-bundle-2.0. | 74 KB     |            |   0% 
cq-parts-bundle-2.0. | 74 KB     | ##1        |  22% 
cq-parts-bundle-2.0. | 74 KB     | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done

things seem to work fine (so far),

- EXCEPT -

the original performance issue above:

simple parametric 15x15 grid union takes 3 minutes on 3 GHz CPU :-)

is still there, only now it takes 2 minutes on 12x12 grid instead :-)

### start
### finish @ 135.34 sec

here is my test, am I missing something obvious?

shared config: arkon.py

#
#
#

import cadquery as cq

wall_gap = 0.5
wall_size = 2.0
lock_size = 3


def export(model:object, file_name:str):
    ""
    with open(file_name, 'w') as file_data:
        cq.exporters.exportShape(model, cq.exporters.ExportTypes.STL, file_data, tolerance=0.01)


class cooler:
    ""
    case_side = 120
    case_depth = 25


class base_inner:
    ""
    side = cooler.case_side + wall_gap
    depth = 30
    floor = wall_size * 1.5


class base_outer:
    ""
    side = base_inner.side + wall_size
    depth = base_inner.depth


class base_stud:
    ""
    side = 12
    spike = 30
    depth = base_outer.depth + spike
    outer_cham = 2
    inner_cham = 4


class base_slit:
    ""
    side = lock_size + wall_gap
    depth = base_stud.spike


class base_grid:
    ""
    border_size = 3
    outer_face = base_inner.side - 2 * (base_stud.side - base_slit.side)
    outer_side = base_stud.spike
    inner_face = outer_face - border_size * 2
    inner_side = outer_side - border_size * 2
    inner_depth = lock_size
    vent_count = 16


class flow_grid:
    ""
    border_size = 2.0
    outer_side = cooler.case_side
    inner_side = outer_side - border_size * 2
    inner_depth = 4.0
    vent_count = 12
    needle_side = 1.0

grid generator: flow_grid.py

#
#
#

import math
import time
import cadquery as cq

from ioner import arkon
from ioner.arkon import flow_grid


def model_flow_grid():
    ""
    count = flow_grid.vent_count

    point_step = flow_grid.inner_side / count
    print(f"point_step: {point_step}")

    cross_side = 2
    cross_depth = flow_grid.inner_depth

    step_init = -(count - 1) / 2
    print(f"step_init: {step_init}")

    point_list = []
    for step_X in range(count):
        for step_Y in range(count):
            point = ((step_init + step_X) * point_step, (step_init + step_Y) * point_step)
            point_list.append(point)

    shape_outer = (cq
        .Workplane("XY")
        .box(flow_grid.outer_side, flow_grid.outer_side, flow_grid.inner_depth)
    )

    shape_inner = (cq
        .Workplane("XY")
        .box(flow_grid.inner_side, flow_grid.inner_side, flow_grid.inner_depth)
    )

    def make_cross(point:tuple) -> cq.Shape:
        print(point)
        flat = cq.Solid.makeBox(cross_side, point_step, cross_depth, point)
        tall = cq.Solid.makeBox(point_step, cross_side, cross_depth, point)
        return flat.fuse(tall)

    frame = shape_outer.cut(shape_inner)

    outer_neck_dia = math.sqrt(2) * (flow_grid.needle_side + arkon.wall_size * 2)
    inner_neck_dia = math.sqrt(2) * (flow_grid.needle_side)

    matrix = (cq
        .Workplane("XY")
#         .transformed(offset=(0, 0, -cross_depth / 2))
        #
        .rarray(point_step, point_step, count, count)
        .box(cross_side, point_step, cross_depth)
        #
        .rarray(point_step, point_step, count, count)
        .box(point_step, cross_side, cross_depth)
        #
        .faces("<Z").workplane()
        .rarray(point_step, point_step, count, count)
        .polygon(4, outer_neck_dia).extrude(-cross_depth)
        #
        .faces("<Z").workplane()
        .rarray(point_step, point_step, count, count)
        .polygon(4, inner_neck_dia).cutThruAll()
    )

    return frame.union(matrix)


if __name__ == "__main__":
    print(f"### start")
    time_start = time.time()
    model_flow = model_flow_grid()
    time_finish = time.time()
    time_diff = time_finish - time_start
    print(f"### finish @ {time_diff:.2f} sec")
    model_flow.exportSvg("flow_grid.svg")
    arkon.export(model_flow, "flow_grid.stl")

therefore a question:

do you have similar tests somewhere?
which work in seconds, not minutes?
can you please share these tests?

thank you

jmwright · 2020-04-06T11:14:21Z

@Andrei-Pozolotin

do you have similar tests somewhere?

Nothing like the grid you have created.

Almost all the time seems to be spent on the matrix creation just before return frame.union(matrix).

Andrei-Pozolotin · 2020-04-06T15:08:31Z

all the time seems to be spent on the matrix

thank you. any expert advice on that union() ?

adam-urbanczyk · 2020-04-06T15:09:38Z

@Andrei-Pozolotin take a look at #312, it should improve the union speed even without 7.4.

Andrei-Pozolotin · 2020-04-06T15:37:36Z

take a look at #312,

outstanding, thank you.
could you perhaps also port that / sync master into:
https://github.com/CadQuery/cadquery/tree/OCP

adam-urbanczyk · 2020-04-06T15:40:48Z

What's the speedup factor BTW?

Andrei-Pozolotin · 2020-04-06T15:59:54Z

speedup factor

you mean old-oce-vs-new-ocp?

I am not sure exactly - test itself was changing in the process
I did not bother to measure exactly, since the order of delay (minutes) is a no-go
I promise I will do proper measurement for old-oce-vs-new-ocp after Bool op refactoring #312 available

adam-urbanczyk · 2020-04-10T16:04:05Z

I merged master to OCP again @Andrei-Pozolotin

adam-urbanczyk · 2020-04-24T16:33:50Z

Any updates on the speed story? #312 is in the OCP branch and conda packages for OCP are available form the cadquery channel BTW.

adam-urbanczyk · 2020-06-10T15:58:55Z

Master is using OCP now so I'm going to close this. One user reported 6x speed improvements, but it might not be relevant for the use case of OP.

tester-occ-7.4.0

c4e7925

This was referenced Mar 20, 2020

Investigate PythonOCC based on OCCT7.4 #226

Closed

PROJECT FUTURE cqparts/cqparts#159

Closed

adam-urbanczyk closed this Jun 10, 2020

tester-occ-7.4.0 #300

tester-occ-7.4.0 #300

Uh oh!

Conversation

Andrei-Pozolotin commented Mar 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmwright commented Mar 21, 2020

Uh oh!

jmwright commented Mar 21, 2020

Uh oh!

Andrei-Pozolotin commented Mar 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Andrei-Pozolotin commented Mar 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jmwright commented Mar 22, 2020

Uh oh!

adam-urbanczyk commented Mar 22, 2020

Uh oh!

Andrei-Pozolotin commented Mar 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adam-urbanczyk commented Mar 22, 2020

Uh oh!

Andrei-Pozolotin commented Mar 22, 2020

Uh oh!

adam-urbanczyk commented Mar 26, 2020

Uh oh!

Andrei-Pozolotin commented Mar 26, 2020

Uh oh!

Andrei-Pozolotin commented Apr 4, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

still facing a performance problem after switch to cq2/ocp

Uh oh!

jmwright commented Apr 6, 2020

Uh oh!

Andrei-Pozolotin commented Apr 6, 2020

Uh oh!

adam-urbanczyk commented Apr 6, 2020

Uh oh!

Andrei-Pozolotin commented Apr 6, 2020

Uh oh!

adam-urbanczyk commented Apr 6, 2020

Uh oh!

Andrei-Pozolotin commented Apr 6, 2020

Uh oh!

adam-urbanczyk commented Apr 10, 2020

Uh oh!

adam-urbanczyk commented Apr 24, 2020

Uh oh!

adam-urbanczyk commented Jun 10, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Andrei-Pozolotin commented Mar 20, 2020 •

edited

Loading

Andrei-Pozolotin commented Mar 21, 2020 •

edited

Loading

Andrei-Pozolotin commented Mar 21, 2020 •

edited

Loading

Andrei-Pozolotin commented Mar 22, 2020 •

edited

Loading

Andrei-Pozolotin commented Apr 4, 2020 •

edited

Loading

still facing a performance problem after switch to `cq2/ocp`