Fix SCHEMA DROP CASCADE with continuous aggregates #2560

pmwkaa · 2020-10-16T17:36:30Z

This change fixes the situation when schema object is dropped
before the cagg which leads to an error when it tries to
resolve it.

Fixes: #2350

codecov · 2020-10-16T17:49:32Z

Codecov Report

Merging #2560 (1b5d1c7) into master (0e57992) will increase coverage by 0.09%.
The diff coverage is 92.26%.

@@            Coverage Diff             @@
##           master    #2560      +/-   ##
==========================================
+ Coverage   90.03%   90.13%   +0.09%     
==========================================
  Files         212      212              
  Lines       34304    34286      -18     
==========================================
+ Hits        30885    30903      +18     
+ Misses       3419     3383      -36

Impacted Files	Coverage Δ
src/guc.c	`97.22% <ø> (ø)`
src/time_utils.c	`97.50% <ø> (ø)`
tsl/src/fdw/option.c	`42.18% <ø> (+9.37%)`	⬆️
tsl/src/fdw/scan_plan.c	`92.89% <ø> (ø)`
tsl/src/remote/data_fetcher.h	`100.00% <ø> (ø)`
tsl/src/remote/row_by_row_fetcher.c	`71.42% <62.50%> (-4.77%)`	⬇️
tsl/src/fdw/relinfo.c	`96.47% <70.58%> (+0.66%)`	⬆️
tsl/src/remote/cursor_fetcher.c	`72.89% <83.33%> (+2.58%)`	⬆️
tsl/src/remote/async.c	`86.43% <85.00%> (-2.42%)`	⬇️
src/bgw_policy/policy.c	`96.42% <94.44%> (-3.58%)`	⬇️
... and 22 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fbf412f...1b5d1c7. Read the comment docs.

svenklemm · 2020-10-19T08:16:48Z

tsl/test/expected/continuous_aggs_ddl.out

+--
+-- Issue: #2350
+CREATE SCHEMA my_cool_schema;
+SET search_path to my_cool_schema, public; -- timescaledb is created in the public schema


please skip the search_path messing and specify the schema directly in the create statements:
CREATE TABLE my_cool_schema.telemetry_raw ...
SELECT create_hypertable('my_cool_schema.telemetry_raw' ...
CREATE MATERIALIZED VIEW my_cool_schema.telemetry_1s ...

or if you think search_path interaction is worth testing have extra tests that test it in addition to specifying schema directly

Sure, just thought it makes sense to keep it since it is the same case as in the issue description

gayyappan · 2020-10-19T17:58:33Z

src/continuous_agg.c

+				ca.relid = get_relname_relid(NameStr(data->user_view_name), nspid);
+			else
+				ca.relid = InvalidOid;
+			ca.data = *data;


shouldn't these 2 lines be conditional? i.e.
i.e. this is valid only if you can find a view object.

Suggested change

ca.data = *data;

if( OidIsValid(...) )

{

ca.relid = ...

ca.data = *data;

drop_continuous_agg(...)

}

Yes, you right. Missed that. But it should not be used anyway I believe since the relation marked as not found.

erimatnor

I think we should try to hook into DROP SCHEMA before it is executed by PostgreSQL so that we can drop dependent objects while they are still valid.

I think it is problematic to have functions that can create both fully valid objects (e.g., Hypertable) and "partial" objects (without all fields set) because it is hard to know which version of the object is produced for a particular case.

erimatnor · 2020-10-20T07:50:27Z

src/continuous_agg.c

@@ -577,7 +577,15 @@ ts_continuous_agg_drop_hypertable_callback(int32 hypertable_id)

 		if (data->raw_hypertable_id == hypertable_id)
 		{
-			continuous_agg_init(&ca, data);
+			/* schema might not exists at this point */


Suggested change

/* schema might not exists at this point */

/* schema might already be dropped at this point */

erimatnor · 2020-10-20T07:55:48Z

src/continuous_agg.c

+			/* schema might not exists at this point */
+			Oid nspid = get_namespace_oid(NameStr(data->user_view_schema), true);
+
+			if (OidIsValid(nspid))


This doesn't seem like the right approach to me. Unless I misunderstand what is going on here, it seems like the schema could already have been dropped at this point, and therefore we accept that we cannot know the relid and will create a cagg object without the proper relids set.

I think that a better way to do this is to make sure that we hook into DROP SCHEMA prior to the call being processed by PostgreSQL and perform the cleanup of our objects at that point when the relations still exist. I believe that is how we deal with many dependency drops in other cases.

I understand your concern. I tried to make this fix as smaller as possible because we already have a lot of assumption that this functionality can be called during CASCADE which expects some of the objects to be deleted: https://github.com/timescale/timescaledb/blob/master/src/continuous_agg.c#L459. Otherwise this will require more changes and reworking of the drop path I believe, basically as you suggested.

Well, those assumptions aren't great either, TBH.

I am actually most concerned about this PR introducing an ambiguous state for the Hypertable object, in a change below. We've had issues related to such things before where some fields could be set or not set, and we've fixed most of them to always be consistent. This change would move us back towards a situation where objects can again be in different "states".

erimatnor · 2020-10-20T07:58:06Z

src/hypertable.c

+	if (OidIsValid(namespace_oid))
+		h->main_table_relid = get_relname_relid(NameStr(h->fd.table_name), namespace_oid);
+	else
+		h->main_table_relid = InvalidOid;


Allowing the creation of Hypertable objects without a valid relid I think is problematic, in particular since this function makes no distinction between the cases the object is created for (i.e., valid use or drop).

I think it is safer to only allow creation of a fully "valid" object or have it fail. Allowing the creation of objects with different state is often a source of subtle errors.

pmwkaa · 2020-10-27T15:22:13Z

drop_continuous_agg() refactored and now tries to work with catalog data only assuming that depending objects (including associated schema) might not exists anymore. Also seems like drop_user_view argument was never used before, fixed that.

k-rus · 2020-10-29T10:31:57Z

@pmwkaa It seems that you have a linking issue on Windows. From Appveyor log:

  timescaledb-tests.vcxproj -> C:\projects\timescaledb\build\test\src\timescaledb-tests.dir\Debug\timescaledb-tests.lib
550     Creating library C:/projects/timescaledb/build/src/Debug/timescaledb-2.1.0-dev.lib and object C:/projects/timescaledb/build/src/Debug/timescaledb-2.1.0-dev.exp
551continuous_agg.obj : error LNK2001: unresolved external symbol InvalidObjectAddress [C:\projects\timescaledb\build\src\timescaledb.vcxproj]
552C:\projects\timescaledb\build\src\Debug\timescaledb-2.1.0-dev.dll : fatal error LNK1120: 1 unresolved externals [C:\projects\timescaledb\build\src\timescaledb.vcxproj]

k-rus · 2020-10-29T10:33:14Z

src/continuous_agg.c

+	ObjectAddress user_view = InvalidObjectAddress;
+	ObjectAddress partial_view = InvalidObjectAddress;
+	ObjectAddress direct_view = InvalidObjectAddress;
+	ObjectAddress rawht_trig = InvalidObjectAddress;
+	ObjectAddress raw_hypertable = InvalidObjectAddress;
+	ObjectAddress mat_hypertable = InvalidObjectAddress;


I guess these lines introduce the linking error on Windows.

k-rus

I have a comment about the signature helper functions. See the comments.

k-rus · 2020-11-02T08:12:05Z

src/continuous_agg.c

@@ -405,37 +405,64 @@ ts_continuous_agg_find_by_rv(const RangeVar *rv)
 	return ts_continuous_agg_find_by_relid(relid);
 }

+static void
+get_rel_by_name(ObjectAddress *addr, const char *schema, const char *name, LOCKMODE mode)


This signature is unintuitive and not easy to use. Usually the variable to output is either the last in the signature or is returned by the function, which is preferable. Cannot you return it? I believe C compiler make sure that the struct will be allocated in the caller.

Also updated the argument types to avoid Name casting

k-rus · 2020-11-02T08:12:28Z

src/continuous_agg.c

+}
+
+static void
+get_rel_by_hypertable_id(ObjectAddress *addr, int32 hypertable_id, LOCKMODE mode)


Same comment, it is better to return or at least the output is the last.

k-rus · 2020-11-02T08:14:28Z

src/continuous_agg.c

+ *
+ * These objects are: the user view itself, the catalog entry in continuous-agg,
+ * the partial view, the materialization hypertable, trigger on the raw hypertable
+ * (hypertable specified in the user view) copy of the user view query


Suggested change

* (hypertable specified in the user view) copy of the user view query

* (hypertable specified in the user view), copy of the user view query

k-rus · 2020-11-02T08:20:18Z

src/continuous_agg.c

-	/* AccessExclusiveLock is needed to drop this table. */
-	LockRelationOid(mat_hypertable->main_table_relid, AccessExclusiveLock);
+	/* NOTE: the lock order matters, see tsl/src/materialization.c.
+	 * Perform all locking upfront


Suggested change

* Perform all locking upfront

* Perform all locking upfront.

k-rus · 2020-11-02T08:20:34Z

src/continuous_agg.c

-		/* raw hypertable is locked above */
-		LockRelationOid(rawht_trigoid, AccessExclusiveLock);
+		/* The trigger will be dropped if the hypertable still exists and no other
+		 * caggs attached */


Suggested change

* caggs attached */

* caggs attached. */

src/continuous_agg.c

gayyappan · 2020-11-02T14:01:34Z

src/continuous_agg.c

@@ -405,37 +405,68 @@ ts_continuous_agg_find_by_rv(const RangeVar *rv)
 	return ts_continuous_agg_find_by_relid(relid);
 }

+static ObjectAddress
+get_rel_by_name(Name schema, Name name, LOCKMODE mode)


Rename as lock_rel_by_name as that is the primary function here?

I don't think this is accurate, because the relation might not exists. In such case semantically it is more close to the get (or get and lock if exists) operation and it returns its address too. Maybe get_and_lock_rel_by_name? not fan of long names

yes, get_and_lock_rel_by_name sounds good.

gayyappan · 2020-11-02T14:02:10Z

src/continuous_agg.c

+}
+
+static ObjectAddress
+get_rel_by_hypertable_id(int32 hypertable_id, LOCKMODE mode)


Suggested change

get_rel_by_hypertable_id(int32 hypertable_id, LOCKMODE mode)

lock_rel_by_hypertable_id(int32 hypertable_id, LOCKMODE mode)

could you rename this as well?

gayyappan · 2020-11-02T14:04:37Z

src/continuous_agg.c

-		LockRelationOid(user_view.objectId, AccessExclusiveLock);
-
-	raw_hypertable = ts_hypertable_get_by_id(agg->data.raw_hypertable_id);
-	/* The raw hypertable might be already dropped if this is a cascade from that drop */


These comments about locking and situtaions related to dependent objects being dropped, should be retained. Otherwise, it is not clear why it is ok to not throw an error if the object is not found.

I've updated the function description in order to describe the situation. There were couple of duplicate messages about the locking, not sure how much value they have. I can put them back, if you would like

I think it is still good to keep these comments about locking and objects that could have been potentially dropped (line 459 , 463 , 464 and 500)..

gayyappan · 2020-11-02T14:06:35Z

src/continuous_agg.c

-		LockRelationOid(direct_view.objectId, AccessExclusiveLock);
-
-	/*  END OF LOCKING. Perform actual deletions now. */
+	partial_view = get_rel_by_name(&cadata->partial_view_schema,


Please retain the comment at line 500 (from the old code)

gayyappan · 2020-11-02T14:12:25Z

tsl/test/sql/continuous_aggs_ddl.sql

+
+SELECT create_hypertable('test_schema.telemetry_raw', 'ts');
+
+CREATE MATERIALIZED VIEW test_schema.telemetry_1s


could you add 1 more mat view for the test case so that we have multiple caggs attached to the hypertable?

erimatnor · 2020-11-02T14:14:20Z

src/continuous_agg.c

@@ -405,37 +405,68 @@ ts_continuous_agg_find_by_rv(const RangeVar *rv)
 	return ts_continuous_agg_find_by_relid(relid);
 }

+static ObjectAddress
+get_rel_by_name(Name schema, Name name, LOCKMODE mode)


Is it possible to make objects const?

Suggested change

get_rel_by_name(Name schema, Name name, LOCKMODE mode)

get_rel_by_name(const Name schema, const Name name, LOCKMODE mode)

erimatnor · 2020-11-02T14:19:05Z

src/continuous_agg.c

 *
 * drop_user_view indicates whether to drop the user view.
 *                (should be false if called as part of the drop-user-view callback)
 */
 static void
-drop_continuous_agg(ContinuousAgg *agg, bool drop_user_view)
+drop_continuous_agg(FormData_continuous_agg *cadata, bool drop_user_view)


I think this function is a bit overloaded and also has a misleading name. Ideally we should better distinguish between dropping the metadata (which is not done in this function) and related internal objects that implement the cagg (tables, views, etc.), and dependent objects that the drop "cascades" to (e.g., jobs). This function does a little bit of everything and it makes it harder to follow.

To fix this, the function might need further refactoring, which is not necessary for this PR. But I think a better function name might be:

Suggested change

drop_continuous_agg(FormData_continuous_agg *cadata, bool drop_user_view)

drop_continuous_agg_objects(FormData_continuous_agg *cadata, bool drop_user_view)

The comment should also be update to reflect this (i.e., that only the objects, such as tables and views are dropped, and not the metadata).

erimatnor · 2020-11-02T14:19:48Z

src/continuous_agg.c

 *
 * drop_user_view indicates whether to drop the user view.
 *                (should be false if called as part of the drop-user-view callback)
 */
 static void
-drop_continuous_agg(ContinuousAgg *agg, bool drop_user_view)
+drop_continuous_agg(FormData_continuous_agg *cadata, bool drop_user_view)
 {
 	ScanIterator iterator =


I suggest moving this initialization to where the iterator is used.

I can do it, but was worried if that has something to do with the locking order

erimatnor · 2020-11-03T10:31:21Z

src/continuous_agg.c

+
+	if (OidIsValid(mat_hypertable.objectId))
+	{
+		performDeletion(&mat_hypertable, DROP_CASCADE, 0);


Why does this need CASCADE, while others don't? It would be good to at least explain this.

I don't have a good explanation for that, rather then it was there before. I can only imaging some user-case depending on this object

CASCADE is required there, it will not work with RESTRICT. Likely for the hypertable it is expected to have depenent objects? (chunks, etc)

I don't think cascade is required normally when dropping a hypertable. This looks potentially dangerous, but I guess it is mitigated by this being an internal table.

erimatnor · 2020-11-03T10:36:14Z

src/continuous_agg.c

+ * which implies that most of the dependent objects potentially could be
+ * dropped including associated schema.
+ *
+ * These objects are: the user view itself, the catalog entry in continuous-agg,


I think this would be better described with a bullet-list of all objects.

erimatnor · 2020-11-03T10:36:34Z

src/continuous_agg.c

+ * These objects are: the user view itself, the catalog entry in continuous-agg,
+ * the partial view, the materialization hypertable, trigger on the raw hypertable
+ * (hypertable specified in the user view), copy of the user view query
+ * (aka the direct view).


Suggested change

* (aka the direct view).

* (AKA the direct view).

erimatnor · 2020-11-03T10:38:13Z

tsl/test/sql/continuous_aggs_ddl.sql

+
+DROP SCHEMA test_schema CASCADE;
+
+SELECT true FROM pg_catalog.pg_namespace WHERE nspname = 'test_schema';


Can you show that the internal objects (e.g., the materialized hypertable and views) are also removed from the internal namespaces (or whatever namespaces they existed in).

erimatnor

One comment nit.

erimatnor · 2020-11-09T13:46:05Z

src/continuous_agg.c

- * trigger on the raw hypertable (hypertable specified in the user view )
- * copy of the user view query (aka the direct view)
- * NOTE: The order in which the objects are dropped should be EXACTLY the same as in materialize.c"
+ * This function is intedended to be run by event trigger during CASCADE,


Suggested change

* This function is intedended to be run by event trigger during CASCADE,

* This function is intended to be run by event trigger during CASCADE,

erimatnor · 2020-11-09T13:48:56Z

src/continuous_agg.c

+
+	if (OidIsValid(mat_hypertable.objectId))
+	{
+		performDeletion(&mat_hypertable, DROP_CASCADE, 0);


I don't think cascade is required normally when dropping a hypertable. This looks potentially dangerous, but I guess it is mitigated by this being an internal table.

gayyappan

Changes look good. Request to retain additional comments.

This change fixes the situation when schema object is dropped before the cagg which leads to an error when it tries to resolve it. Issue: timescale#2350

This release candidate contains bugfixes since the previous release candidate, as well as additional minor features including support for "user-mapping" authentication between access/data nodes and an experimental API for refreshing continuous aggregates on individual chunks. **Minor Features** * timescale#2627 Add optional user mappings support * timescale#2635 Add API to refresh continuous aggregate on chunk **Bugfixes** * timescale#2560 Fix SCHEMA DROP CASCADE with continuous aggregates * timescale#2593 Set explicitly all lock parameters in alter_job * timescale#2604 Fix chunk creation on hypertables with foreign key constraints * timescale#2610 Support analyze of internal compression table * timescale#2612 Optimize internal cagg_watermark function * timescale#2613 Refresh correct partial during refresh on drop * timescale#2617 Fix validation of available extensions on data node * timescale#2619 Fix segfault in decompress_chunk for chunks with dropped columns * timescale#2620 Fix DROP CASCADE for continuous aggregate * timescale#2625 Fix subquery errors when using AsyncAppend * timescale#2626 Fix incorrect total_table_pages setting for compressed scan * timescale#2628 Stop recursion in cache invalidation

This release candidate contains bugfixes since the previous release candidate, as well as additional minor features including support for "user-mapping" authentication between access/data nodes and an experimental API for refreshing continuous aggregates on individual chunks. **Minor Features** * #2627 Add optional user mappings support * #2635 Add API to refresh continuous aggregate on chunk **Bugfixes** * #2560 Fix SCHEMA DROP CASCADE with continuous aggregates * #2593 Set explicitly all lock parameters in alter_job * #2604 Fix chunk creation on hypertables with foreign key constraints * #2610 Support analyze of internal compression table * #2612 Optimize internal cagg_watermark function * #2613 Refresh correct partial during refresh on drop * #2617 Fix validation of available extensions on data node * #2619 Fix segfault in decompress_chunk for chunks with dropped columns * #2620 Fix DROP CASCADE for continuous aggregate * #2625 Fix subquery errors when using AsyncAppend * #2626 Fix incorrect total_table_pages setting for compressed scan * #2628 Stop recursion in cache invalidation

pmwkaa requested a review from a team as a code owner October 16, 2020 17:36

pmwkaa requested review from k-rus, WireBaron, svenklemm and a team and removed request for a team October 16, 2020 17:36

svenklemm reviewed Oct 19, 2020

View reviewed changes

svenklemm added this to the 2.0.0 milestone Oct 19, 2020

svenklemm approved these changes Oct 19, 2020

View reviewed changes

pmwkaa force-pushed the cagg_drop_schema_fix branch from c0045f6 to cc588de Compare October 19, 2020 11:32

pmwkaa requested a review from a team October 19, 2020 13:32

gayyappan reviewed Oct 19, 2020

View reviewed changes

erimatnor requested changes Oct 20, 2020

View reviewed changes

pmwkaa force-pushed the cagg_drop_schema_fix branch 2 times, most recently from d19c765 to 52b1410 Compare October 27, 2020 15:10

pmwkaa requested review from gayyappan, erimatnor and a team October 27, 2020 15:22

k-rus reviewed Oct 29, 2020

View reviewed changes

pmwkaa force-pushed the cagg_drop_schema_fix branch 3 times, most recently from a53e73c to 4199b75 Compare October 29, 2020 11:54

pmwkaa requested a review from k-rus October 29, 2020 12:12

k-rus reviewed Nov 2, 2020

View reviewed changes

pmwkaa force-pushed the cagg_drop_schema_fix branch from 4199b75 to 8235ee5 Compare November 2, 2020 12:04

pmwkaa requested a review from k-rus November 2, 2020 12:05

gayyappan reviewed Nov 2, 2020

View reviewed changes

erimatnor reviewed Nov 3, 2020

View reviewed changes

pmwkaa linked an issue Nov 4, 2020 that may be closed by this pull request

Drop continuous aggregate with cascade fails. #2608

Closed

pmwkaa removed a link to an issue Nov 6, 2020

Drop continuous aggregate with cascade fails. #2608

Closed

pmwkaa force-pushed the cagg_drop_schema_fix branch 2 times, most recently from 3da0ec7 to 88201c1 Compare November 9, 2020 13:17

pmwkaa requested review from erimatnor and gayyappan November 9, 2020 13:18

erimatnor approved these changes Nov 9, 2020

View reviewed changes

gayyappan approved these changes Nov 9, 2020

View reviewed changes

pmwkaa force-pushed the cagg_drop_schema_fix branch from 88201c1 to e212809 Compare November 10, 2020 08:22

Fix SCHEMA DROP CASCADE with continuous aggregates

1b5d1c7

This change fixes the situation when schema object is dropped before the cagg which leads to an error when it tries to resolve it. Issue: timescale#2350

pmwkaa force-pushed the cagg_drop_schema_fix branch from e212809 to 1b5d1c7 Compare November 10, 2020 08:29

pmwkaa merged commit bbcb2b2 into timescale:master Nov 10, 2020

pmwkaa mentioned this pull request Nov 12, 2020

Release 2.0.0-rc3 #2637

Merged

-			ca.data = *data;
+if( OidIsValid(...) )
+{
+            ca.relid = ...
+			ca.data = *data;
+			drop_continuous_agg(...)
+}

	/* schema might not exists at this point */
	/* schema might already be dropped at this point */

	* (hypertable specified in the user view) copy of the user view query
	* (hypertable specified in the user view), copy of the user view query

	get_rel_by_hypertable_id(int32 hypertable_id, LOCKMODE mode)
	lock_rel_by_hypertable_id(int32 hypertable_id, LOCKMODE mode)


		SELECT create_hypertable('test_schema.telemetry_raw', 'ts');

		CREATE MATERIALIZED VIEW test_schema.telemetry_1s

	get_rel_by_name(Name schema, Name name, LOCKMODE mode)
	get_rel_by_name(const Name schema, const Name name, LOCKMODE mode)

	drop_continuous_agg(FormData_continuous_agg *cadata, bool drop_user_view)
	drop_continuous_agg_objects(FormData_continuous_agg *cadata, bool drop_user_view)


		DROP SCHEMA test_schema CASCADE;

		SELECT true FROM pg_catalog.pg_namespace WHERE nspname = 'test_schema';

	* This function is intedended to be run by event trigger during CASCADE,
	* This function is intended to be run by event trigger during CASCADE,

Fix SCHEMA DROP CASCADE with continuous aggregates #2560

Fix SCHEMA DROP CASCADE with continuous aggregates #2560

Conversation

pmwkaa commented Oct 16, 2020 • edited by svenklemm

codecov bot commented Oct 16, 2020 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gayyappan Oct 19, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor Oct 20, 2020 • edited

Choose a reason for hiding this comment

pmwkaa Oct 20, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmwkaa commented Oct 27, 2020 • edited

k-rus commented Oct 29, 2020

Choose a reason for hiding this comment

k-rus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gayyappan Nov 2, 2020 • edited

Choose a reason for hiding this comment

pmwkaa Nov 2, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmwkaa Nov 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pmwkaa Nov 3, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gayyappan left a comment

Choose a reason for hiding this comment

pmwkaa commented Oct 16, 2020 •

edited by svenklemm

codecov bot commented Oct 16, 2020 •

edited

gayyappan Oct 19, 2020 •

edited

erimatnor Oct 20, 2020 •

edited

pmwkaa Oct 20, 2020 •

edited

pmwkaa commented Oct 27, 2020 •

edited

gayyappan Nov 2, 2020 •

edited

pmwkaa Nov 2, 2020 •

edited

pmwkaa Nov 3, 2020 •

edited

pmwkaa Nov 3, 2020 •

edited