Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Metrics Page #97

Closed
inngvar opened this issue Sep 17, 2020 · 5 comments
Closed

Support Metrics Page #97

inngvar opened this issue Sep 17, 2020 · 5 comments
Assignees

Comments

@inngvar
Copy link
Contributor

inngvar commented Sep 17, 2020

Is your feature request related to a problem? Please describe.
Jhipster generated interface has /admin/metrics page, but it didn't work

Describe the solution you'd like
Metrics page works

@avdev4j
Copy link
Contributor

avdev4j commented Oct 6, 2020

I'm taking this one

@avdev4j avdev4j self-assigned this Oct 6, 2020
@avdev4j
Copy link
Contributor

avdev4j commented Oct 7, 2020

After a quick study of this feature, there is my opinion. We can't add this feature out of the box just by adding the metrics Quarkus dependency.
Metrics send back the backend application are so far different.

JHipster Spring boot metrics

{
  "jvm" : {
    "G1 Old Gen" : {
      "committed" : 9.961472E7,
      "max" : 4.294967296E9,
      "used" : 2.9753296E7
    },
    "G1 Survivor Space" : {
      "committed" : 1.6777216E7,
      "max" : -1.0,
      "used" : 1.6777216E7
    },
    "CodeHeap 'non-profiled nmethods'" : {
      "committed" : 1.6515072E7,
      "max" : 2.44105216E8,
      "used" : 1.6458368E7
    },
    "Compressed Class Space" : {
      "committed" : 1.4024704E7,
      "max" : 1.073741824E9,
      "used" : 1.2832832E7
    },
    "Metaspace" : {
      "committed" : 9.865216E7,
      "max" : -1.0,
      "used" : 9.53716E7
    },
    "G1 Eden Space" : {
      "committed" : 1.5204352E8,
      "max" : -1.0,
      "used" : 1.42606336E8
    },
    "CodeHeap 'non-nmethods'" : {
      "committed" : 2555904.0,
      "max" : 7553024.0,
      "used" : 1338880.0
    }
  },
  "databases" : {
    "min" : {
      "value" : 10.0
    },
    "idle" : {
      "value" : 10.0
    },
    "max" : {
      "value" : 10.0
    },
    "usage" : {
      "0.0" : 0.0,
      "1.0" : 0.0,
      "max" : 0.0,
      "totalTime" : 56.0,
      "mean" : 14.0,
      "0.5" : 0.0,
      "count" : 4,
      "0.99" : 0.0,
      "0.75" : 0.0,
      "0.95" : 0.0
    },
    "pending" : {
      "value" : 0.0
    },
    "active" : {
      "value" : 0.0
    },
    "acquire" : {
      "0.0" : 0.0,
      "1.0" : 0.0,
      "max" : 0.0,
      "totalTime" : 0.0,
      "mean" : 0.0,
      "0.5" : 0.0,
      "count" : 4,
      "0.99" : 0.0,
      "0.75" : 0.0,
      "0.95" : 0.0
    },
    "creation" : {
      "0.0" : 0.0,
      "1.0" : 0.0,
      "max" : 0.0,
      "totalTime" : 0.0,
      "mean" : 0.0,
      "0.5" : 0.0,
      "count" : 0,
      "0.99" : 0.0,
      "0.75" : 0.0,
      "0.95" : 0.0
    },
    "connections" : {
      "value" : 10.0
    }
  },
  "http.server.requests" : {
    "all" : {
      "count" : 21
    },
    "percode" : {
      "200" : {
        "max" : 0.0,
        "mean" : 46.15621905,
        "count" : 20
      },
      "401" : {
        "max" : 0.0,
        "mean" : 72.563256,
        "count" : 1
      }
    }
  },
  "cache" : {
    "usersByEmail" : {
      "cache.gets.miss" : 0.0,
      "cache.puts" : 0.0,
      "cache.gets.hit" : 0.0,
      "cache.removals" : 0.0,
      "cache.evictions" : 0.0
    },
    "com.mycompany.myapp.domain.User" : {
      "cache.gets.miss" : 0.0,
      "cache.puts" : 1.0,
      "cache.gets.hit" : 0.0,
      "cache.removals" : 0.0,
      "cache.evictions" : 0.0
    },
    "usersByLogin" : {
      "cache.gets.miss" : 1.0,
      "cache.puts" : 1.0,
      "cache.gets.hit" : 1.0,
      "cache.removals" : 0.0,
      "cache.evictions" : 0.0
    },
    "com.mycompany.myapp.domain.Authority" : {
      "cache.gets.miss" : 0.0,
      "cache.puts" : 2.0,
      "cache.gets.hit" : 0.0,
      "cache.removals" : 0.0,
      "cache.evictions" : 0.0
    },
    "com.mycompany.myapp.domain.User.authorities" : {
      "cache.gets.miss" : 0.0,
      "cache.puts" : 1.0,
      "cache.gets.hit" : 0.0,
      "cache.removals" : 0.0,
      "cache.evictions" : 0.0
    }
  },
  "garbageCollector" : {
    "jvm.gc.max.data.size" : 4.294967296E9,
    "jvm.gc.pause" : {
      "0.0" : 0.0,
      "1.0" : 0.0,
      "max" : 0.0,
      "totalTime" : 7.0,
      "mean" : 7.0,
      "0.5" : 0.0,
      "count" : 1,
      "0.99" : 0.0,
      "0.75" : 0.0,
      "0.95" : 0.0
    },
    "jvm.gc.memory.promoted" : 2.3031216E7,
    "jvm.gc.memory.allocated" : 9.09115392E8,
    "classesLoaded" : 18224.0,
    "jvm.gc.live.data.size" : 2.9753296E7,
    "classesUnloaded" : 0.0
  },
  "services" : {
    "/management/info" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 40.807746,
        "count" : 1
      }
    },
    "/management/jhimetrics" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 25.177456,
        "count" : 1
      }
    },
    "/management/threaddump" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 97.601243,
        "count" : 1
      }
    },
    "/api/authenticate" : {
      "POST" : {
        "max" : 0.0,
        "mean" : 408.096317,
        "count" : 1
      }
    },
    "root" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 72.563256,
        "count" : 1
      }
    },
    "/api/account" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 41.063897,
        "count" : 1
      }
    },
    "/**" : {
      "GET" : {
        "max" : 0.0,
        "mean" : 20.691848133333334,
        "count" : 15
      }
    }
  },
  "processMetrics" : {
    "system.load.average.1m" : 5.11962890625,
    "system.cpu.usage" : 0.08736286062576189,
    "process.start.time" : 1.602059160607E12,
    "system.cpu.count" : 12.0,
    "process.files.open" : 217.0,
    "process.cpu.usage" : 0.002836509586847543,
    "process.uptime" : 48756.0,
    "process.files.max" : 10240.0
  }
}

Quarkus metrics

{
    "base": {
        "cpu.systemLoadAverage": 1.5263671875,
        "thread.count": 51,
        "classloader.loadedClasses.count": 15431,
        "classloader.unloadedClasses.total": 0,
        "gc.total;name=G1 Young Generation": 11,
        "jvm.uptime": 393594,
        "thread.max.count": 75,
        "memory.committedHeap": 268435456,
        "classloader.loadedClasses.total": 15432,
        "cpu.availableProcessors": 12,
        "thread.daemon.count": 19,
        "gc.total;name=G1 Old Generation": 0,
        "memory.maxHeap": 4294967296,
        "cpu.processCpuLoad": 0.0,
        "gc.time;name=G1 Old Generation": 0,
        "memory.usedHeap": 166137472,
        "gc.time;name=G1 Young Generation": 87
    },
    "vendor": {
        "memoryPool.usage.max;name=G1 Survivor Space": 20971520,
        "memory.freePhysicalSize": 81379328,
        "memoryPool.usage.max;name=CodeHeap 'non-profiled nmethods'": 14720000,
        "memoryPool.usage;name=Metaspace": 88871544,
        "memoryPool.usage;name=G1 Eden Space": 0,
        "memoryPool.usage;name=CodeHeap 'non-profiled nmethods'": 14732800,
        "memoryPool.usage;name=G1 Old Gen": 0,
        "memoryPool.usage.max;name=CodeHeap 'non-nmethods'": 1393920,
        "memoryPool.usage.max;name=G1 Old Gen": 60253568,
        "cpu.processCpuTime": 11152983000,
        "memory.committedNonHeap": 120360960,
        "memoryPool.usage.max;name=Compressed Class Space": 11083672,
        "memoryPool.usage.max;name=G1 Eden Space": 158334976,
        "memory.freeSwapSize": 1314914304,
        "memoryPool.usage.max;name=Metaspace": 88870744,
        "cpu.systemCpuLoad": 0.0,
        "memory.usedNonHeap": 116082336,
        "memoryPool.usage;name=CodeHeap 'non-nmethods'": 1393920,
        "memoryPool.usage;name=G1 Survivor Space": 9437184,
        "memoryPool.usage;name=Compressed Class Space": 11083672,
        "memory.maxNonHeap": -1
    },
    "application": {
    }
}

Quarkus metrics micrometer compatibility enabled

{
    "base": {
        "jvm.threads.states;state=new": 0,
        "jvm.memory.max;area=nonheap;id=Metaspace": -1,
        "jvm.buffer.count;id=mapped": 0,
        "jvm.memory.committed;area=heap;id=G1 Eden Space": 136314880,
        "jvm.buffer.total.capacity;id=mapped": 0,
        "jvm.memory.used;area=nonheap;id=Compressed Class Space": 11126576,
        "jvm.memory.committed;area=heap;id=G1 Survivor Space": 2097152,
        "jvm.memory.max;area=nonheap;id=Compressed Class Space": 1073741824,
        "jvm.threads.states;state=runnable": 29,
        "jvm.memory.used;area=nonheap;id=CodeHeap 'non-profiled nmethods'": 14207104,
        "jvm.threads.states;state=timed-waiting": 12,
        "jvm.memory.used;area=nonheap;id=Metaspace": 89223392,
        "jvm.gc.memory.allocated": 0,
        "jvm.classes.loaded": 15488,
        "jvm.classes.unloaded": 0,
        "jvm.memory.committed;area=nonheap;id=CodeHeap 'non-nmethods'": 2555904,
        "jvm.threads.daemon": 24,
        "jvm.memory.committed;area=nonheap;id=Metaspace": 91570176,
        "jvm.memory.used;area=heap;id=G1 Old Gen": 72787968,
        "jvm.memory.max;area=heap;id=G1 Survivor Space": -1,
        "jvm.memory.used;area=heap;id=G1 Survivor Space": 2097152,
        "jvm.buffer.count;id=direct": 33,
        "jvm.memory.used;area=nonheap;id=CodeHeap 'non-nmethods'": 1391360,
        "jvm.memory.committed;area=heap;id=G1 Old Gen": 130023424,
        "jvm.gc.max.data.size": 0,
        "process.start.time": 1602058496042,
        "jvm.memory.max;area=heap;id=G1 Old Gen": 4294967296,
        "jvm.memory.max;area=nonheap;id=CodeHeap 'non-nmethods'": 7553024,
        "jvm.memory.max;area=nonheap;id=CodeHeap 'non-profiled nmethods'": 244105216,
        "jvm.threads.peak": 65,
        "jvm.gc.live.data.size": 0,
        "jvm.threads.live": 56,
        "jvm.threads.states;state=blocked": 0,
        "jvm.memory.committed;area=nonheap;id=CodeHeap 'non-profiled nmethods'": 14221312,
        "jvm.gc.memory.promoted": 0,
        "jvm.buffer.memory.used;id=direct": 322317,
        "jvm.memory.used;area=heap;id=G1 Eden Space": 35651584,
        "jvm.buffer.memory.used;id=mapped": 0,
        "jvm.memory.max;area=heap;id=G1 Eden Space": -1,
        "process.runtime": 80562,
        "jvm.buffer.total.capacity;id=direct": 322315,
        "jvm.memory.committed;area=nonheap;id=Compressed Class Space": 12058624,
        "jvm.threads.states;state=terminated": 0,
        "jvm.threads.states;state=waiting": 15
    },
    "vendor": {
    },
    "application": {
    }
}

Another solution could be to use the micrometer metrics (This is used by JHipster) but the endpoint return only text plain content:

# HELP process_files_open_files The open file descriptor count
# TYPE process_files_open_files gauge
process_files_open_files 408.0
# HELP system_cpu_count The number of processors available to the Java virtual machine
# TYPE system_cpu_count gauge
system_cpu_count 12.0
# HELP jvm_buffer_count_buffers An estimate of the number of buffers in the pool
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 29.0
# HELP jvm_memory_usage_after_gc_percent The percentage of old gen heap used after the last GC event, in the range [0..1]
# TYPE jvm_memory_usage_after_gc_percent gauge
jvm_memory_usage_after_gc_percent{area="heap",generation="old",} 0.0
# HELP process_files_max_files The maximum file descriptor count
# TYPE process_files_max_files gauge
process_files_max_files 10240.0
# HELP jvm_classes_unloaded_classes_total The total number of classes unloaded since the Java virtual machine has started execution
# TYPE jvm_classes_unloaded_classes_total counter
jvm_classes_unloaded_classes_total 0.0
# HELP jvm_memory_used_bytes The amount of used memory
# TYPE jvm_memory_used_bytes gauge
jvm_memory_used_bytes{area="heap",id="G1 Survivor Space",} 1.048576E7
jvm_memory_used_bytes{area="heap",id="G1 Old Gen",} 6.143176E7
jvm_memory_used_bytes{area="nonheap",id="Metaspace",} 9.043236E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 1401728.0
jvm_memory_used_bytes{area="heap",id="G1 Eden Space",} 9.7517568E7
jvm_memory_used_bytes{area="nonheap",id="Compressed Class Space",} 1.1341312E7
jvm_memory_used_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1.4614016E7
# HELP jvm_memory_max_bytes The maximum amount of memory in bytes that can be used for memory management
# TYPE jvm_memory_max_bytes gauge
jvm_memory_max_bytes{area="heap",id="G1 Survivor Space",} -1.0
jvm_memory_max_bytes{area="heap",id="G1 Old Gen",} 4.294967296E9
jvm_memory_max_bytes{area="nonheap",id="Metaspace",} -1.0
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 7553024.0
jvm_memory_max_bytes{area="heap",id="G1 Eden Space",} -1.0
jvm_memory_max_bytes{area="nonheap",id="Compressed Class Space",} 1.073741824E9
jvm_memory_max_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 2.44105216E8
# HELP jvm_buffer_memory_used_bytes An estimate of the memory that the Java virtual machine is using for this buffer pool
# TYPE jvm_buffer_memory_used_bytes gauge
jvm_buffer_memory_used_bytes{id="mapped",} 0.0
jvm_buffer_memory_used_bytes{id="direct",} 276512.0
# HELP process_start_time_seconds Start time of the process since unix epoch.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.602082376702E9
# HELP jvm_gc_overhead_percent An approximation of the percent of CPU time used by GC activities over the last lookback period or since monitoring began, whichever is shorter, in the range [0..1]
# TYPE jvm_gc_overhead_percent gauge
jvm_gc_overhead_percent 0.0
# HELP jvm_threads_daemon_threads The current number of live daemon threads
# TYPE jvm_threads_daemon_threads gauge
jvm_threads_daemon_threads 22.0
# HELP http_server_connections_seconds_max  
# TYPE http_server_connections_seconds_max gauge
http_server_connections_seconds_max 83.981719394
# HELP http_server_connections_seconds  
# TYPE http_server_connections_seconds summary
http_server_connections_seconds_active_count 3.0
http_server_connections_seconds_duration_sum 230.472197978
# HELP http_server_bytes_written_max  
# TYPE http_server_bytes_written_max gauge
http_server_bytes_written_max 36590.0
# HELP http_server_bytes_written  
# TYPE http_server_bytes_written summary
http_server_bytes_written_count 14.0
http_server_bytes_written_sum 78105.0
# HELP jvm_threads_live_threads The current number of live threads including both daemon and non-daemon threads
# TYPE jvm_threads_live_threads gauge
jvm_threads_live_threads 58.0
# HELP jvm_classes_loaded_classes The number of classes that are currently loaded in the Java virtual machine
# TYPE jvm_classes_loaded_classes gauge
jvm_classes_loaded_classes 15824.0
# HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
# TYPE process_cpu_usage gauge
process_cpu_usage 2.1301343963061486E-4
# HELP jvm_threads_peak_threads The peak live thread count since the Java virtual machine started or peak was reset
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 77.0
# HELP jvm_memory_committed_bytes The amount of memory in bytes that is committed for the Java virtual machine to use
# TYPE jvm_memory_committed_bytes gauge
jvm_memory_committed_bytes{area="heap",id="G1 Survivor Space",} 1.048576E7
jvm_memory_committed_bytes{area="heap",id="G1 Old Gen",} 1.21634816E8
jvm_memory_committed_bytes{area="nonheap",id="Metaspace",} 9.306112E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-nmethods'",} 2555904.0
jvm_memory_committed_bytes{area="heap",id="G1 Eden Space",} 1.3631488E8
jvm_memory_committed_bytes{area="nonheap",id="Compressed Class Space",} 1.2320768E7
jvm_memory_committed_bytes{area="nonheap",id="CodeHeap 'non-profiled nmethods'",} 1.4680064E7
# HELP http_server_requests_seconds  
# TYPE http_server_requests_seconds summary
http_server_requests_seconds_count{method="POST",outcome="SUCCESS",status="200",uri="/api/authenticate",} 3.0
http_server_requests_seconds_sum{method="POST",outcome="SUCCESS",status="200",uri="/api/authenticate",} 0.785574197
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/management/info",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/management/info",} 0.05630222
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/api/account",} 3.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/api/account",} 5.09574647
http_server_requests_seconds_count{method="GET",outcome="CLIENT_ERROR",status="404",uri="NOT_FOUND",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="CLIENT_ERROR",status="404",uri="NOT_FOUND",} 0.119552748
http_server_requests_seconds_count{method="GET",outcome="CLIENT_ERROR",status="401",uri="/api/account",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="CLIENT_ERROR",status="401",uri="/api/account",} 0.014083244
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/manifest.webapp",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/manifest.webapp",} 0.010005822
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/app/8.chunk.js",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/app/8.chunk.js",} 0.009773769
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/app/6.chunk.js",} 1.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/app/6.chunk.js",} 0.003817346
http_server_requests_seconds_count{method="GET",outcome="SUCCESS",status="200",uri="/management/jhimetrics",} 2.0
http_server_requests_seconds_sum{method="GET",outcome="SUCCESS",status="200",uri="/management/jhimetrics",} 0.030323012
# HELP http_server_requests_seconds_max  
# TYPE http_server_requests_seconds_max gauge
http_server_requests_seconds_max{method="POST",outcome="SUCCESS",status="200",uri="/api/authenticate",} 0.444617552
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/management/info",} 0.05630222
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/api/account",} 5.084611755
http_server_requests_seconds_max{method="GET",outcome="CLIENT_ERROR",status="404",uri="NOT_FOUND",} 0.119552748
http_server_requests_seconds_max{method="GET",outcome="CLIENT_ERROR",status="401",uri="/api/account",} 0.014083244
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/manifest.webapp",} 0.010005822
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/app/8.chunk.js",} 0.009773769
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/app/6.chunk.js",} 0.003817346
http_server_requests_seconds_max{method="GET",outcome="SUCCESS",status="200",uri="/management/jhimetrics",} 0.017930907
# HELP http_server_bytes_read  
# TYPE http_server_bytes_read summary
http_server_bytes_read_count 15.0
http_server_bytes_read_sum 173.0
# HELP http_server_bytes_read_max  
# TYPE http_server_bytes_read_max gauge
http_server_bytes_read_max 58.0
# HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
# TYPE jvm_buffer_total_capacity_bytes gauge
jvm_buffer_total_capacity_bytes{id="mapped",} 0.0
jvm_buffer_total_capacity_bytes{id="direct",} 276510.0
# HELP jvm_info JVM version info
# TYPE jvm_info gauge
jvm_info{runtime="OpenJDK Runtime Environment",vendor="AdoptOpenJDK",version="11.0.7+10",} 1.0
# HELP jvm_threads_states_threads The current number of threads having NEW state
# TYPE jvm_threads_states_threads gauge
jvm_threads_states_threads{state="runnable",} 28.0
jvm_threads_states_threads{state="blocked",} 0.0
jvm_threads_states_threads{state="waiting",} 21.0
jvm_threads_states_threads{state="timed-waiting",} 9.0
jvm_threads_states_threads{state="new",} 0.0
jvm_threads_states_threads{state="terminated",} 0.0
# HELP system_cpu_usage The "recent cpu usage" for the whole system
# TYPE system_cpu_usage gauge
system_cpu_usage 0.07057572783774943
# HELP process_uptime_seconds The uptime of the Java virtual machine
# TYPE process_uptime_seconds gauge
process_uptime_seconds 113.332
# HELP system_load_average_1m The sum of the number of runnable entities queued to available processors and the number of runnable entities running on the available processors averaged over a period of time
# TYPE system_load_average_1m gauge
system_load_average_1m 1.96142578125

I don't know how we should handle that. Rewrite a whole part of the frontend part seems not to be the best idea since we want to avoid to handle too many modifications for only one feature.
We can just enable metrics but not modify the frontend for the moment (useful in case we want to deploy a microservice and monitor metrics through a registry).
For the frontend we can manipulate the result to have an expected JSON but I don't know if worth it.

I'll open a new PR to add micrometer with a default registry dependency.

WDYT @joewhite101 @danielpetisme ?

@danielpetisme
Copy link
Member

I think micrometer is the good option.
Maybe it worth share the point with the upstream jhipster project. IMHO, use micrometer as standard interface rather than having a Spring oriented page could make sense.
WDYT?

@avdev4j
Copy link
Contributor

avdev4j commented Oct 8, 2020

Maybe, we should wait. There is a chance that this kind of admin pages will no longer to the generator itself -> jhipster/generator-jhipster#12090

We should suggest modifications to JHCC directly, WDYT?

@avdev4j avdev4j closed this as completed Dec 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants